« Return to Blog

An explanation about the server troubles this week

7/4/2015 in ClearCheckbook News
First off, we want to say we're extremely sorry for all the trouble the site and app has had this week. We received thousands of emails during the outages and there's no way to easily respond to everyone. If you contacted us to express your confusion or frustration, we're sorry we didn't get back with you. We spent that time working on getting the site back up and running.

The site went down because our database has been getting overloaded lately. Imagine each page load as someone walking through a turnstyle to enter a subway or stadium. Under normal circumstances the lines move through normally as everyone goes through one at a time. Now, imagine rush hour when there are many more people trying to go through those same set of turnstlyes. Things start to back up and people get impatient. In the web world, this leads to people refreshing the page which just makes the situation worse since each page load causes more and more requests to the database (essentially adding more and more people in line). This snowballs and eventually the wait time is so long and there are so many requests pending that the server doesn't know how to handle it so it starts throwing errors.

The main culprit is our Transactions table in the database. We have somewhere around 75 million transactions that our users have entered with somewhere around 50,000 new transactions added daily. This is a lot of data to constantly look up, add to and delete from. The hardware that we're currently running right now was built and set up around 2008 and the truth is, I think the site has gotten too big for our hardware to handle. Back then, we had outgrown several other hosting solutions and at the time, cloud based hosting was still in its infancy. We purchased hardware that we thought would last us for a long time, and has done so pretty well for the last 7 or so years.

For an immediate fix, we've backed up all transactions and are having you restore them when you log in by clicking a button at the top of the page. This process could take a few minutes but don't worry, your data is still secured and will be restored shortly (there are a lot of other people trying to do the same thing).

To help prevent these downtimes and database problems from happening in the future, we're working with an outside party to help us migrate into newer hosting solutions that aren't bound by a physical machine (namely, cloud based solutions that can expand with demand). This is the next logical step in our site's growth, but it's not something we want to jump into blindly. We're going to do our research and find the best solution to keep ClearCheckbook running smoothly for years to come.

If you're accessing the site through one of the mobile apps, you'll need to log into the website to manually restore your transactions for now. We'll work on updating this soon though.

Again, I can't express the depth of sorrow, stress and frustration I feel that we've had these issues. To compound on it, the 4th of July holiday weekend in the US has made it extremely hard to find outside help since everyone is out on vacation.

We'll keep you updated with the progress of the migration to newer hardware over the next week.

Brandon O'Brien
Founder of ClearCheckbook

Comments for this post:

No worries
Thanks a bunch for the update, Brandon. I think I speak for a lot of users when I say that I still love this site and will stick with it. If it means enduring occasional growing pains as the site grows larger and more popular, then so be it. Keep your chin up, take a breather, and then move onward. Will look ahead to your updates on the data migration :)
by Sky_21 on Jul 4, 2015
Thanks
Thanks for working through this Brandon!
by 51t0fccb on Jul 4, 2015
Transactions not restoring
What do we do if our transactions are not restoring after we hit the button on the site?
by cspence12 on Jul 4, 2015
A testimony to this site's awesomeness.
The fact that this site has been growing so fast in popularity is just a testimony as to how good it is.
by superfunguy.old.1436229517 on Jul 4, 2015
Thank You
Thank You guys for your hard work on getting things running again. Love this site! Hope you can enjoy your 4th!!
by jbstauffer on Jul 4, 2015
Hang In There... and Thanks.
As an IT professional, I have worked for several companies over the years that have had issues like these at the worst possible time. So I have been there and I feel for you. And yes, not much worse than on a long holiday weekend - yuk! This is one of the few times I have been on "this side" of it, so I understand. Hang in there! Glad to hear you are not "rushing" into something - I've seen that too - where a bigger mistake is made or one that just prolongs the pain. Take your time, get it right, and like others have said, this isn't all bad - it's a testament to the popularity of your site. As a die-hard CCB user, I'm not going anywhere either, so enjoy what little you can of the holiday and we look forward to a bright future.
by jdemmerich on Jul 4, 2015
Thanks
Keep up the great work Brandon. I'd still like to see the apps work without Internet and just refresh when the connection was restored but that's something I'm sure you'll consider in time. You have a great app, site, and core values and I recommend your product to all my family and friends.
by hellokittie1950 on Jul 4, 2015
Downtime
I feel your pain ... Good luck going forward!
by dmcgee on Jul 4, 2015
Restore does not work for many users
Brandon, would you be willing to add some more instructions for people who have repeatedly tried to restore within the past 24 hours but without success? Any other steps to take, or bail to paper and pencil? Your product has been very valuable to me for years and I thank you for your work.
by raiyagar on Jul 4, 2015
Restore Not Working
I agree with raiyagar. Love the product and will stick with it. However, even after using the restore button, my transactions are only showing up through January of this year. What should I try next? Thanks for the assistance!!
by illdoitagain on Jul 4, 2015
Missing large chunk of my data
I have been using ClearCheckbook since 2008. When I restore, I only get back a year or so of information, from an account that I no longer use. I'm hoping this is due to traffic on the site, and not the only information that was saved. Should I attempt to retrieve my information repeatedly or is it all lost?
by Struiksma on Jul 4, 2015
Only a small step back
There have been many steps forward in your development, I am sure that I will see more to come. As long as you will take the positive things out of this failure, the risk for future incidents to take place will decrease.

Keep up the good work. CCB is an excellent service, thank you!
by RazvanBC on Jul 5, 2015
Thank you and no worries
Thanks, Brandon, for all the hard work you and your team have done especially over this holiday weekend to get this site back up and running. This is one of my favorite web services to use and this outage has in no way changed my opinion about your site. Keep up the great work!!
by fkornre on Jul 5, 2015
Restoring not working.
I understand that this is an issue you are working diligently to fix. Thank you for that. I would like you to be clear, however, regarding the likelihood that data has been lost. I have tried to restore 5 times, and usually it hasn't worked at all. This time it restored some 2011 data. Thank you.
by YMSBLS on Jul 5, 2015
Restoration Non-responsive
Glad to know you are working to restore service, I know it must be frustrating. The restore feature is not reliable and only restoring 2013 transactions and the mobile app is non-responsive at all. Thanks for working to fix the glitch.
by kenroberts1 on Jul 5, 2015
My transactions appear to be lost forever??
Not a single transaction has been restored. Even after I follow your instructions. Clearly, nothing is fixed.
by lavdasg on Jul 5, 2015
URGENT!!! Restore is not working at all!!
I have gone through your restore process 3 times and I have no transactions at all!! Please tell me how to get this corrected ASAP, I'm flying blind right now.
by pctech8633 on Jul 5, 2015
Thanks for update Brandon
For those still missing trans/categories....i have been here since 09'...thats 6 years of trans. I hit the restore button at the top of the page 7 times and also cleared my cache before EVERYTHING finally came back. So keep trying if you have not succeeded yet. I am a huge supporter of ccb so not going anywhere. Im not running a business off this app, so it is for me just an inconvenience, not what could be a nuclear breakdown for some that are trying to run a business using ccb.
by PHishHead on Jul 5, 2015
Patience to those trying to restore their data.
Hi everyone, I thought I had lost all of my data and a couple of attempts to restore yesterday was not successful. Am pleased to say my data has successfully restored this evening 8:52pm UK time. So hang on in there!

Best wishes from across the pond to all CC uses.
Ps to Brandon and the team thanks for keeping us informed and good luck I hope you manage to get the issues resolved.
by accentra on Jul 5, 2015
Dont' worry !
Hi Brandon, thanks for your work and don't worry it happens and everybody learn from their own mistakes.
Good luck !
by vittoriop77 on Jul 5, 2015
Missing Transactions
I am still missing data after following the restore process. It restored transactions up until 2013 only. How do I restore the transactions all the way up to the current date before the outage?
by kevinleclair on Jul 5, 2015
Update: Missing Transactions
All of my transactions appear to be loaded now - thank you for your hard work on this great website/service!
by kevinleclair on Jul 5, 2015
Still have your back.
If I could translate downtime into cash, I'm still way ahead judging by the volume of overdrafts I've avoided since I've used your site going back over six years. I will keep trying the restoring option. All accounts and categories are restored but so far no transactions.
by acook4357 on Jul 5, 2015
no transactions
Hi I cannot restore my transactions. Help please.
by carrieann71 on Jul 5, 2015
All My Data Still Missing
My data is still missing. I've done the restore several times. The only information is from 2007. Please help
by cdbecker4 on Jul 5, 2015
Thanks
Thank you for keeping us informed. It helps a lot. Take care and have a blessed day!
by Sonja06 on Jul 5, 2015
Data salad and years of bookeeping down the drain
My data did not restore after Clearcheckbook server malfunction. I looked for help on this website, but there was none. I emailed for help, but no response. I tried to restore my data from my "backup" (which is just an export file), but that was a fail as there were many errors with transaction splits. Now there is no option to restore my data. I attempted to use the export files to migrate to several other apps, but that was a fail too. That's years of effort maintaining my books wasted. Also, lots of time wasted this weekend trying to deal with this myself. I think it's sad that there is no real back up and proper restore function with Clearcheckbook. I also think it is inexcusable that the developer did not anticipate server problems well before they happened.
by Wally38 on Jul 6, 2015
Database
Being a System Engineer I understand the problem. What I need to know is when do you expect the repair process and move to the cloud completed and when it will be stable again. I am currently missing about 4 months of transactions. Having to due a similar repair of a MS SQL database system I understand the pressure. I am not planning on going any where your service has been awesome and I am sure it will get much better one the system is back up.
by bburgess on Jul 6, 2015
Restore issue
Brandon, I sincerely appreciate your Service and am with you at this difficult juncture.

I tried restore option as well, but it ended up restroring some transactions more than an year old.

I had been a CCB user, since 2008 and had numerous transactions registered. Not sure if this was the reason for Restore not working for me. I feel lost, since CCB was my powerful Online Ledger all through these years. Even before I login to my different bank accounts, I login to CCB to keep track of my different Accounts all from a single App.

My only prayer...CCB, please come back! Help us restore all our transactions. We really miss you!
by radkrishg3 on Jul 6, 2015
I, too, and am missing transactions
I've hit the "restore" button many times (along with the recommendation of a previous poster to clear my cookies) but still am missing transactions after 6/14. An ETA for resolution would be great and maybe an explanation why the restores are incomplete for many.
by samstearns on Jul 6, 2015
Still Missing Trans???
As I stated earlier, i did Restore 7 times. Every time I did it more & more trans appeared each time....so keep hitting restore & they all should still come back. Mine are back all the way to the start of time in 2009, so nothing lost.
by PHishHead on Jul 6, 2015
Restore Function Just Not Working for Me
Brandon,
by Imfletch on Jul 6, 2015
Restore Function Just Not Working for Me
Brandon, I can appreciate just how overwhelming all this has been for you. And I'm sure you understand how scarey it is for those of us who stopped keeping paper records years ago and suddenly cannot seem to get the system to restore properly. Can you say just a few words to ease our fears on this matter. I've lost count of how many times I've hit the "restore" button, but no matter how many times, the most recent transaction I get is from July 31 of 2014 !! If you can just ease us a little and reassure us that all that missing data really does exist somewhere, we'll hang in there with you because, after all, Clear Checkbook is pretty darned awesome! So, please, just a few words about data that is not restoring. - Jim F.
by Imfletch on Jul 6, 2015
Could you please answer some questions
I understand there is an issue and growth is great. I love CCB and have used it for years. It seems everyone is having the same issues we are only getting partial transactions restored. We have spent several hours trying to restore transactions, with no luck. So now what, please just let us know. Do we start over? Is it good to continue to hit the restore button? Do we all just need to stop trying for a day or two for this issue to be resolved if so put a page up that just says so instead of making us all wait and wonder whats going on.
by stephens on Jul 6, 2015
Thank You
I will be patient as long as it takes. I don't ever want to leave this website! Thank you for the diligent work you are doing on correcting the problem and for your careful and thoughtful procedures. Hang in there!
by Dietzy3 on Jul 6, 2015
Chiming in
Just wanted to second the other comments - the restore only restored partially for me, up to July 2010, with no transactions after that.

Hope there is a way to restore to ~7/1/15 data, after the dust settles....
by dasx86 on Jul 6, 2015
Thanks
I appreciate your work and dedication to finding an excellent solution. My service restored very well. I did not want to even consider another program, so thank you!
by jksca on Jul 6, 2015
Update?
Can you give an update as to whether our data will be recovered? I don't have any information after 2007. I understand that you busy trying to fix your problem, but it would be nice to know whether we are going to get our data back or if we are screwed.
by cdbecker4 on Jul 6, 2015
Update??
What is the status of the recovery and when can we expect our data to be restored?!?
by poopbrown on Jul 6, 2015
Thanks for being diligent in getting things back up
Hey Brandon, I'm an IT guy and I totally get what happened. Scaling an app for growth is challenging, especially when you don't really know what kind of growth you're going to see. The good thing is that you have a good product, and hopefully a majority of patient customers. Thanks again for getting things back up and running.
by paulbahr on Jul 6, 2015
No worries
Thank you for all your hard work, i know its going to take a few days to get everything back up. I havent kept paper records for almost 10 years because of this site, and have no question you'll get it back up. I know you're working hard and it will soon be back up to 100%. Thank you for everything you do and for making an awesome site.
by rvalentine25 on Jul 6, 2015
Still a big fan....
Seems to have stabilized for me... fingers crossed... I lived in IT for a long time and dealt with databases, I feel your pain and the intense pressure to get it operational again. I am sure you will loose a few users, but I am sure you will gain double the loss when you are up and running at full strength. Just please take the time to get it right.
by zerch911 on Jul 6, 2015
Thank You so much!!!
Clear Checkbook is awesome and has made doing our finances so easy! Thank you so much for this great program and no worries on the server issues as these things are bound to happen from time to time. Thank you again for all you do!!
by kverdi on Jul 7, 2015
Thanks
Thanks for taking a bit of time and putting out a "OK, this is what's happening update", it helps a lot with understanding what's going on when things start happening on our end.
by djbingham5113229 on Jul 7, 2015
Thank you Brandon!
I am sorry to say that I am one of those who sent an email of frustration. I apologize for that; I let my impatience and annoyance at being inconvenienced get the better of me. I hope you never do get around to reading all those emails and that mine is one of those you miss. Not that I resorted to name calling or anything of that nature but I vented and I am very sorry. I really do love CCB! I kept my checkbook balanced for decades without any issue until this year. I changed banks at the beginning of the year and for some reason had not been able to reconcile....until I signed up to CCB. I guess that is where part of the frustration came from. Years of keeping my checkbook balanced, then all the confusion and frustration because I couldn't once I changed banks and right when I thought I had the solution and everything was good again THIS happens.
CBB is absolutely wonderful and I hope you get the issue resolved soon. I will be a user for years to come. Thanks for that great post. It was very informative and it helped to know we (I) am not just a customer but you do care enough - even though you are dealing with anxiety and frustration yourself in all this - to take the time to let us know what is going on.
by clynch56 on Jul 7, 2015
Update Please
When can we expect to see the site operational again. It would be really great to my account balanced and know what the balance is so I can spend money again.
by bburgess on Jul 7, 2015
Update Please
Like all of the other users I think an update on the issue would be great. The silence has me fearing for the worse...that my transactions are gone. Knowing if they are gone or not would atleast help me to move on to the next best solution, which may be another provider, and I would hate to do that since I have been using CCB since 2007.
by mlmarsh1 on Jul 7, 2015
Update Please
An update on the matter would be greatly appreciated by myself and many others. I have multiple accounts with no transaction history from the beginning of the year and also upcoming/future transactions that I have forecasted about a year out. Knowing whether this data is completely lost would help me decide in what step to take next, whether I need to 1.) Re-enter large amounts of data/transactions into the system myself, 2.) Wait a while longer and the data will re-appear, 3.) Switch providers.
by Daniel_B on Jul 8, 2015
Yes, an update would be nice...
I did a restore and unfortunately I still cannot view all of my transactions and all but one of my accounts are gone.
by Lh21774 on Jul 8, 2015
7/8/2015 8:11am
The restore is still not working for me. I have sent 2 emails and have gotten no response. I am flying blind and really need a check book. All I see are my April transactions. Help!
by jscott123 on Jul 8, 2015
Need a update
I am premium member I love your site but this is not costing me money and time. If I do not soon get this working I will be forced to change providers.
by bburgess on Jul 8, 2015
If you're really sorry, you'll keep your equipment up to date so this doesn't happen again.
It's a rookie mistake.
by wiese67 on Jul 8, 2015

Archives

» 2018
» 2017
» 2016
» 2015
» 2014
» 2013
» 2012
» 2011
» 2010
» 2009
» 2008
» 2007
» 2006

Categories

Contact ClearCheckbook

If you have a question about the site, please check the Support Forums, Tutorials and Help section to see if it's already answered there.

Your message has been sent.

Thank you for reaching out to us. We work hard to respond as quickly as possible to our emails. You'll receive a response soon.