Building Your Systems for Christmas and Easter
I was chatting with a customer the other day about their Alfresco implementation and trying to better understand both what they were doing currently and what they hoped to achieve in 2014. During the conversation, they discussed a current project to increase the scale of their current infrastructure. As I thought through the process, a familiar phrase ran through my head.
You don’t build a church for Christmas or Easter.
As I mulled that thought over, I realized that when it came to systems, that it didn’t apply.
Building for the Everyday
I don’t recall where I first heard the phrase when designing systems, but I do recall the point. When building a church, you don’t design it to handle the crowds during Christmas and Easter. You design it to handle your steady congregation with some room for growth, say 15-20%. The reason is that for 50 weeks out of the year, there will be a lot of wasted resources spent on utilities and maintenance in order to accommodate a crowd that would likely contribute the same amount regardless. You also lose that sense of community when half of the sanctuary is empty on the average Sunday.
This premise applies to stores or other institutions that have such peak demands. A store should be designed to serve the community and handle the payday shopping crowds. It should not be designed to handle the pre-snow storm rush. The store, like the church, will make it through that peak demand without losing its regular visitors.
Of course, this maxim falls apart in the land of business systems.
When architecting a system, a common approach is to take the average daily peak activity, add any appropriate growth factor for the life of the hardware, and then add 20% to account for both the margin of error and surprise peaks. Hardware and software are expensive, so scaling unnecessarily is something to be avoided, even in this age of virtual environments.
This approach doesn’t work for business critical systems or systems that have cyclical peaks, like the ones churches do around Christmas and Easter.
Think about an eCommerce site. It has peak traffic in December. The multiple layers of infrastructure cannot be designed for the other 11 months. It has to be designed to handle the December sales rush. If the site goes down, people leave and never come back. Money is lost every minute. Even a slowdown will lead to lost sales.
Think about internal systems. How busy is the CRM system at the end of a quarter, especially at the end of the fiscal year? It is a significant jump as deals are coming in under the wire. If those systems are not working, then the deals may not be recorded.
During the call with our customer, I was running math through my head.
- What is the current capacity?
- What is the average daily load?
- What is the anticipated increase in users, processes, and information in percentage?
- Take the increase, add 20% to anticipate future growth, multiply it against the daily load, determine by what percentage it exceeds capacity, and increase capacity by that amount.
Except that is exactly what they should NOT be doing. They publish content on a scheduled basis. They have peak periods during publishing and have to hit deadlines. While they could likely tolerate a slight slowdown, they can’t have any significant delays.
What they need to do is start by determining the peak usage and use that as a basis for the increase, not the average daily load. If that isn’t available and there hasn’t been significant performance issues to-date, then using the current capacity is a good starting point.
For business critical, you have to design for Christmas and Easter. If you don’t, there won’t be any visitors left by the next holiday season.