Transactions fully restored and more clarity about what happened

7/9/2015 in ClearCheckbook News

All of your transactions have been restored to your accounts and they will show up as they normally do on both the web and mobile apps.

What the heck happened?
Now, a little more explanation about what happened and what we're doing to future-proof the site. On the morning of Tuesday June 30th, the database started getting hung up on certain queries which caused a queue of queries to form. When this happens, the server tries its best to get through the queue as fast as possible, but in doing so, overloads and causes the queue to essentially stop in its tracks. This makes itself visible to you by seeing a bunch of errors and not being able to log into the site.

We were able to get things running again for the rest of the day but two days later on the morning of Thursday July 2nd the database had the same issue. We scrambled to get things running again but everything we'd done in the past simply wasn't working. We narrowed the problem down to a single table in our database, the Transactions table. This is what stores all of the transactions that ClearCheckbook users add to the site, all 75+ million of them. Whenever changes are made to this table (adding, editing or deleting transactions), indexes that help the database run more efficiently are updated. Normally this is a quick process but on 75M rows of data, this can take a long time.

To try and speed up the reading/writing of this table, we archived transactions for users who hadn't logged into the site for a few months. Unfortunately this didn't help too much and after a few hours we were back to the database overloading issues. By now it's Friday July 3rd and we're scrambling to get things figured out but it seems like nothing we do is helping at all. It's now that we make some calls and, fortunately, even on a Friday afternoon before Independence Day in the United States, we were able to reach some outside help that was able to meet up on Saturday July 4th and work with us.

The first thing we did was backup all transactions and force manual restoring when you logged back in. This worked fairly well and at least got the site operational. The problem was many people were reporting that not all of their transactions fully restored and were concerned that their data was lost.

What we're doing in the short term
This obviously wasn't a long term solution and the confusion that arose made us speed up plans to break up the giant Transactions table into about 30 smaller tables based on user id. This will be our mid-to-long term solution as far as the database schema goes. The downtime tonight was necessary so we could move all the transactions from the giant tables into the smaller ones. In the short term, this will help the site maintain uptime on our current hardware since the database won't have to search 75M transactions regularly.

What's next
The next step, and what we're working on now, is migrating to a cloud based solution. The outside help we mentioned earlier also excels at making these transitions and has done so with other companies in the past. We're working with them now to optimize our site code for its next home in the cloud. The cloud solution will help us expand and scale up as the site grows without having to maintain our own servers.

A heartfelt apology
I have to admit that this is the single most stressful situation I've ever been in before. When I took ClearCheckbook on as my full time job back in 2009, I knew there would be growing challenges along the way. I use the site daily myself for both personal and business accounts. When the site is having problems, I know how frustrating it can be.

What compounds the frustration is when I receive threatening, cursing and hate filled messages to my personal cell phone and email accounts and our social media outlets and the ClearCheckbook blog while we're working our hardest to resolve the problems. At the height of the issues, we were receiving several hundred emails an hour. There's no possible way we could respond to each one of your messages individually and still have time to work on the database problems.

I can't express how sorry I am that this happened the way it did. Please know that we do have a plan of action to prevent this kind of mess from happening in the future and I'll be posting more information about it as a plan and timeline solidifies.

If you're a premium member who contacted us to cancel/refund your membership, we tried to search through all the messages and fulfill your requests, but there are probably some that slipped through the cracks. If you haven't heard back from us and still wish to stop using the site, please contact us again through the Contact Us link at the bottom right side of the page.

Again, from the bottom of my heart, I'm so extremely sorry for the inconveniences the downtime caused you.

Brandon OBrien
Founder, ClearCheckbook

Update about restoring transactions

7/8/2015 in ClearCheckbook News
We're finishing up some updates that will restore everyone's transactions without having to click the restore button. This will also fix any issues that some people are having where it looks like a subset of your transactions is still missing even after clicking the restore button. These updates will be pushed out by the end of the day today.

Again, we're sorry we can't respond to every email coming in. If you're concerned that all of your data is lost, it isn't. This confusion will be resolved after we make the above updates live.

An explanation about the server troubles this week

7/4/2015 in ClearCheckbook News
First off, we want to say we're extremely sorry for all the trouble the site and app has had this week. We received thousands of emails during the outages and there's no way to easily respond to everyone. If you contacted us to express your confusion or frustration, we're sorry we didn't get back with you. We spent that time working on getting the site back up and running.

The site went down because our database has been getting overloaded lately. Imagine each page load as someone walking through a turnstyle to enter a subway or stadium. Under normal circumstances the lines move through normally as everyone goes through one at a time. Now, imagine rush hour when there are many more people trying to go through those same set of turnstlyes. Things start to back up and people get impatient. In the web world, this leads to people refreshing the page which just makes the situation worse since each page load causes more and more requests to the database (essentially adding more and more people in line). This snowballs and eventually the wait time is so long and there are so many requests pending that the server doesn't know how to handle it so it starts throwing errors.

The main culprit is our Transactions table in the database. We have somewhere around 75 million transactions that our users have entered with somewhere around 50,000 new transactions added daily. This is a lot of data to constantly look up, add to and delete from. The hardware that we're currently running right now was built and set up around 2008 and the truth is, I think the site has gotten too big for our hardware to handle. Back then, we had outgrown several other hosting solutions and at the time, cloud based hosting was still in its infancy. We purchased hardware that we thought would last us for a long time, and has done so pretty well for the last 7 or so years.

For an immediate fix, we've backed up all transactions and are having you restore them when you log in by clicking a button at the top of the page. This process could take a few minutes but don't worry, your data is still secured and will be restored shortly (there are a lot of other people trying to do the same thing).

To help prevent these downtimes and database problems from happening in the future, we're working with an outside party to help us migrate into newer hosting solutions that aren't bound by a physical machine (namely, cloud based solutions that can expand with demand). This is the next logical step in our site's growth, but it's not something we want to jump into blindly. We're going to do our research and find the best solution to keep ClearCheckbook running smoothly for years to come.

If you're accessing the site through one of the mobile apps, you'll need to log into the website to manually restore your transactions for now. We'll work on updating this soon though.

Again, I can't express the depth of sorrow, stress and frustration I feel that we've had these issues. To compound on it, the 4th of July holiday weekend in the US has made it extremely hard to find outside help since everyone is out on vacation.

We'll keep you updated with the progress of the migration to newer hardware over the next week.

Brandon O'Brien
Founder of ClearCheckbook

Introducing Grouped, Combo and Income Budgets plus Including Future Transactions

4/14/2015 in ClearCheckbook Updates

We've been hard at work creating several new updates to the Budgets tool here at ClearCheckbook. Today we're announcing and launching Grouped, Combo and Income budgets as well as the option to include unpaid bills and un-posted scheduled transactions.

We receive suggestions for site improvements all the time and this round of updates represents some of the most frequently requested ideas. This is the biggest update we've ever made to the Budget tool and we're happy to share what these new features will do for you.

New Budget Features!

Grouped Budgets
  • This update gives you the ability to group multiple accounts and categories into a single budget.
    • Grouped Budgets will let you create, for example, one budget that covers both your Checking and Savings accounts but ignores the others. You can also create a grouped budget that covers multiple categories without having to rely on organizing your subcategories.
Combo Budgets
  • Combo Budgets let you create an Account/Category combination.
    • This update will let you create a budget that, for example, covers any expenses made to your Eating Out, Entertainment and Miscellaneous categories but only if they come from your Debit account.
Income Budgets
  • Set a budget goal for how much you want to save and see how you're progressing toward that amount.
    • If you select the Income Budget option, we'll handle the budget a little differently by tracking all of your deposits made to the budgeted account or category and showing you your progress toward the amount goal.
Other Updates
  • An option to Include Future Transactions
    • When this option is enabled we'll search all of your upcoming and unpaid bills from the Bill Tracker as well as any upcoming and un-posted scheduled transactions from the Reminders and Recurring Transactions tool. Any bills or recurring transactions that meet that criteria and fall within the budget duration will be counted toward the budget.

We've added icons for Grouped, Combo and Income Budgets so it's easier to differentiate them in your budget list and budget history reports. These new features are available now for anyone with a ClearCheckbook Premium membership.

Here are some screenshots from the new Budget updates. You can also check out our video tutorial that covers all of the new budgeting features.
New Budget List
Budget List
Combo Budget
Combo Budgets
Future Budget
Future Budget

Are you a University student? Get a free year of ClearCheckbook Premium!

3/26/2015 in ClearCheckbook News
At ClearCheckbook, we strongly believe in the importance of financial literacy and having a good grasp on where your money is going. Student loan debt now tops $1 trillion (averaging over $24,000 per graduate) and is higher than both auto loans and credit card debt. Student loans also have the highest 90+ day delinquency rate out of mortgages, auto loans and credit card debt.

With that in mind, we want to do what we can to help university students have firm control of their finances before graduating. To play our part, we're offering a free year of ClearCheckbook Premium to anyone with a valid .edu email address.

When you register for ClearCheckbook and use a .edu email address, we'll include a confirmation link in your Welcome email. Simply click on that link to activate your premium upgrade. If you're already a member with a .edu email address, you'll see an alert on the Dashboard with a link to send a confirmation email. This free upgrade has no strings attached and no billing information is required.

The only reason for the email link confirmation is to ensure the email address is valid. Per our privacy policy, we won't send you any offers or sell off your email address. We just need to make sure you're actually the owner of the .edu email address.

I started ClearCheckbook when I was in college because I needed a way to manage my own finances. Living on your own and going to school plus all the responsibilities that comes with it can be hard to adjust to at first, but setting budgets and controlling your spending is a great way to set yourself up with a good financial foundation.

Welcome to ClearCheckbook Version 4!

2/21/2015 in ClearCheckbook News

This has been a long time coming but we can finally say that we've launched the next great version of ClearCheckbook! V4 is the culmination of months of hard work spent re-writing code, updating the UI, adding new features, implementing suggestions and a lot of beta testing.

The goal of this update was to keep the site as similar as possible to what you're used to while updating the UI to be more friendly across all platforms, devices and screen sizes. While the site looks different, most of the behavior and locations of tools, links, menus, etc are the same as they were in the last version.

Before I get into all the changes and updates V4 brings... let me first mention a few things:

1. If the site looks weird, graphics aren't loading or you're having problems getting the site to work, please clear the cache in your browser. The easiest way is by refreshing the page about 5-10x in a row really quick.

2. During beta testing, some users complained about issues with contrast between elements on the site. We tracked this down to monitor angle issues or screen color calibration problems. If all the colors seem washed out, try adjusting your monitor angle or running a color calibration on the screen.

Now, on to what's new in V4!

Overall
  • Responsive Design
  • Increased number of autocomplete items from 5 to 10
  • Added 4 new video tutorials for a total of 10 (more will be coming too)
Dashboard
  • Reminders gadget - show alert for recently expired reminders
  • Bills gadget
    • Show alert for recently expired bills
    • Click on a bill to be taken to that bill
  • Adding Gadgets
    • Buttons for gadgets that are already on the dashboard are disabled
    • Not using pop-up box for the add gadgets form
  • Added icons to ClearCheckbooks Links gadget
  • Added 'Other Assets' field to Net Worth gadget to track any additional physical assets
Transaction Register
  • Icons for identifying Transfers and Paid Credit Card transactions
  • Site now remembers your View setting (jived, un-jived, withdrawals, etc)
  • Alternating background colors for jived transactions
  • Temporary account view doesn't get reset on column sort
  • Hover over account, category, description to see full text
  • Hover over sub category to see parent name.
  • Sub categories have a down arrow icon to specify them as sub category
  • When 'Modify Related Transfer' is selected on edit, the related transfer will also update if it's visible on the register
Credit Card section
  • If you change credit card accounts it doesn't reset your view options
Search Form
  • Ability to only show split children/transfers/paid cc transactions
  • Added This Quarter, Last Quarter and Last 90 Days options to date range for premium members
Reports
  • Charts updated and switched from Google Charts API
  • Balances timeline report updates
Debt Snowball
  • Improved display of progress bar
Reminders / Recurring Transactions
  • Improved / standardized navigation between view types
  • Deleting a reminder from the "All" list doesn't take you back to "Upcoming" list
  • Added a border to today's date even if there's a reminder on that day
  • Added a link to reminder emails that lets you disable email reminders
Bill Tracker
  • Improved report viewing
  • Standardized navigation between view types
  • Alert for recently expired bills
Budgets
  • Reports show up for newly created or just edited budgets (rather than having to refresh the page first)
  • Standardized navigation
  • Premium - Export budgets and budget history to CSV file
Budget History
  • Budget history totals are now calculated "on the fly" rather than storing the amount spent at time of reset
    • Provides more accurate historical data in case you added transactions after the budget had reset.
    • No need for the "reset budget history" function
  • View pie chart for spending vs budget on Text history
  • If a budget resets multiple times in a month period, each instance shows up correctly in the monthly text report view
  • Report graphics overhaul
Envelope Budgets
  • If you set a start date for the budget, clicking on the budget will only show transactions on and after that start date
Import Transactions
  • Added ability to click on a transaction to edit it (vs having to select the checkbox and then click the Edit button)
  • Paginating Duplicate Transaction checking
Auto-Jive Tool
  • Added page numbers to list of transactions
Settings
  • Icons to help quickly identify the different settings
Accounts
  • Inactive accounts are now faded
Categories
  • Sub categories in the category list are indented with an icon to help separate them from parent categories

As you can see, there are quite a few updates to check out. This is just the start though. One of the major driving forces behind the V4 update was to better future-proof the site by adopting modern web technologies. This will make it much easier for us to add new features, tools and updates to the site.

The next step for us is focusing on much needed overhauls to all of the mobile apps. We put those on the backburner for a while so we could get the main website updated. Now that V4 has been launched we can go back and spend time on the mobile apps.

ClearCheckbook started out as a tiny program I built and ran on my laptop so I could manage my own finances while I was in college. In the past 9 years ClearCheckbook has gone from something only I could access to a site with over 300,000 registered users managing over 60 million transactions. I just wanted to take a moment to say thanks to all of you for helping ClearCheckbook grow. Whether you've been around since the beginning or just found the site, I'm happy to have built a site that's constantly evolving to fit your financial needs.

Thanks again and enjoy V4!

Brandon O'Brien
Founder, ClearCheckbook.com

ClearCheckbook Reaches 50 Million Transactions

5/5/2014 in ClearCheckbook News
We've passed another milestone today by now storing over 50 million transactions that our users have entered. We're averaging about 28,000 new transactions added each day which is a huge amount!

We're extremely happy to see the site continually growing and we have some great updates planned for the future. We'll post more information about these updates over the next couple of months.

Thanks again for using ClearCheckbook!

You're safe from the Heartbleed SSL exploit

4/9/2014 in ClearCheckbook News
There have been some announcements made in the news today about a new way that SSL connections could be exploited. They're calling this "Heartbleed" and more information can be found here http://heartbleed.com/.

We're running a version of OpenSSL that does not contain the vulnerability. This means that all of your data and information has been and will continue to be safe from SSL exploits while you're using ClearCheckbook.com.

Viewing Multiple Accounts on Transaction Register

1/22/2014 in ClearCheckbook Updates
We just released a new feature, some updates and a few bug fixes. The main new feature is the ability for Premium members to select multiple accounts to view on the Transaction Register. If you're a premium member the Accounts drop down list is now a multi-select form where you can select among any of your active accounts. By selecting multiple accounts your balances will update accordingly and the register will show transactions for each of those accounts combined.

The Search tool was also updated with some new options. When you're searching or exporting transactions you can now select to exclude transfers and paid credit card transactions. This will make it easier to see the correct balances for accounts since you can eliminate duplicates (eg: you paid your credit card with a transfer so now you can hide all the paid transactions since they're covered by that transfer).

In addition to those updates we also knocked out a few small bug fixes related to exporting transactions and the reminders/recurring transactions gadget.

If you're a premium member trying to select multiple accounts and it doesn't seem to be working correctly, please refresh the page about 5-10x in a row really quick to clear your browsers cache.

ClearCheckbook Android App Update

1/3/2014 in ClearCheckbook Updates
We've updated the Android app to version 1.5.0. This update addresses some major freezing/stalling issues when trying to use the application. If you were experiencing any of these issues, please download the update and let us know if you continue to have any problems.

In addition to fixing the stalling issues, we also added the ability to add splits to your transactions from the Add Transactions screen.

The update was submitted to the Play store a little while ago and should be available for download within a couple hours.

Automatic backups of your ClearCheckbook data to Dropbox

11/27/2013 in ClearCheckbook Updates
We're happy to announce the launch of a new ClearCheckbook feature that lets you enable automatic backups of your ClearCheckbook data to your Dropbox account.

If you're a premium member and you have a Dropbox account, you can now enable automatic backups by clicking on Tools at the top of the page, then click on Automatic Backups. Once enabled, your data will be backed up to a CSV file on your Dropbox account each night as long as you've added new transactions since the last backup, have an active premium membership and have the feature enabled.

Contact ClearCheckbook

If you have a question about the site, please check the Support Forums, Tutorials and Help section to see if it's already answered there.
Your message has been sent. We work hard to respond as quickly as possible to our emails. You'll receive a response soon.