Don’t Get Caught With Your App Down on Black Friday & Cyber Monday

As an ecommerce app developer, the holiday season can be the best time of the year with huge numbers of installs coming in – or it can be the worst time of year if things don't go well at crunch time. This year, the National Retail Federation expects online holiday shopping to increase another 11% and even more during big sales days such as Black Friday and Cyber Monday. Is your app ready for it? At Bold, we've done a number of things to ensure a successful weekend that we'd like to share. Here are five of our favorite tips to help your app survive Black Friday and Cyber Monday!

by Jason Myers

Last updated 18 Nov 2014

It's been said that a man's life can be defined by one key moment, one decision, or, maybe most importantly, a lack of a decision. I've come to learn that something similar can be said of an ecommerce company or ecommerce app on Black Friday or Cyber Monday.

As an ecommerce app developer, it can be the best time of the year with huge numbers of installs coming in – or it can be the worst time of year if things don't go well at crunch time.

This year, the National Retail Federation expects online holiday shopping to increase another 11% and even more during big sales days such as Black Friday and Cyber Monday. Is your app ready for it?

At Bold, we've done a number of things to ensure a successful weekend that we'd like to share. Here are five of our favorite tips to help your app survive Black Friday and Cyber Monday!

You might also like: Marketing Your Shopify App: The App Listing Page

Beef up support to the max

Your support team is your lifeblood and will be your front lines against the onslaught of users during Black Friday and Cyber Monday. Whether it's just you doing support by yourself, or you have a full team, do absolutely everything possible to increase availability during this time. Specifically, start the Thursday evening before Black Friday at 11:00 PM Eastern time, right through until the evening of Cyber Monday at about 11:00 PM Pacific. This will be the peak times "beefed up" support is needed.

If things go smoothly and none of your apps experience any outages, you'll still have a higher influx of questions from clients than usual. However, if things don't go well for some reason, it will be 10 times worse if you don't communicate or are not available to answer questions fast. So for better or for worse, it's imperative that you maintain good communication with your customers during peak traffic times like this.

Notify customers what to expect & be real

As developers, we've learned that we need to be somewhat akin to a car mechanic – offering advice to someone about their car before they go on a family road trip. They don't need to know every single little detail, but they do need to know if it can handle the highway and if it will get them where they plan on going. Equally important, they need to know you're behind them and that you'll be available should something go wrong.

This time of year, store owners are not thinking about the possible adverse effects of increased traffic on their shop. In fact, most are quite the opposite and are only thinking about the sales! The things on their minds are what promotions they should be running, making sure they have inventory, staffing, and other things related to their business that they should actually be thinking about!

To allow them to keep focused on their business it's a good idea to send them a quick email letting them know what steps you've take to ensure uptime, extended support, monitoring, and anything else you've done to help ensure their success.

If you have any legitimate concerns related to traffic spikes, it's 100% better to be upfront and honest about it now rather than later. If you have any area of concern, let them know and always give a best-case / worst-case scenario so they can plan accordingly. After all, this is their livelihood we're talking about.

You might also like: Marketing Your Shopify App: The Details We Overlook

Get your servers jacked & monitor them

This point gets a little technical, but it's really the heart of the matter. One of the most important things we did here at Bold was find a hosting company that wanted to help us scale up and could support it. We use Rackspace's managed cloud service and they were also able to help suggest and implement most of the items mentioned below.

This is a list of a few things we implemented to make our apps scalable. We won't go into much detail, but feel free to ask any questions about them in the comments and we'll be happy to answer.

Split up Database, Web, Batch & File storage processes. Doing this ensures that any piece of the application can be scaled up individually. This means that if you’re getting lots of web requests, you can simply add a few extra web servers. Or, if your batch processes are piling up, you can add another server to process the batch queue.
We use a load balancer to split up the requests amongst all web servers.
We use ScaleArc to load balance and add a level of caching to the database level.
In order to keep our web requests snappy, we’ve added a batch level to our applications. In a nutshell, our web servers capture the data and queue up work. Then the batch server will come in and pick up items in the queue to process. We developed our own method to queue and process all the batch requests equally across all of our batch servers.

Important Tip

Many developers struggle with the question of what level of traffic increase to expect. On average, ecommerce stores generally double in traffic during Black Friday and Cyber Monday. The key words here are "on average." Some stores don't increase at all, while others go up drastically.

What we've noticed is that stores that are willing to pay for apps or custom development are usually in the higher echelon of stores, and are also the ones that see a much higher increase. So if you have a customer that is paying for apps, you can bet they're probably going to spike more than two times their normal traffic.

In order to be safe, we've prepared to be able to comfortably handle a 10-time increase in server load, and will have staff on standby all weekend with a plan in place to quickly scale up any of the pieces. The catchphrase around the office when deciding on servers is not, "What can we afford to spend," rather it's, "What can we not afford to lose?"

Along with increasing your server and database capacity, it's important to equally increase your ability to monitor. Here are a few tools that we use for monitoring:

New Relic

New Relic monitors all our servers & apps. It will actually help point out bottlenecks and slow sections in the apps as well. There are alerts you can register for to be informed if your app is experiencing any issues.

New Relic Insights

With the New Relic Insights tool we can see how many requests we're getting, how long we're taking to return files on our server, and how long it takes for the full html pages to render. These insights are invaluable when making server configuration decisions.

ScaleArc

We use ScaleArc to monitor all the Database calls. It reports on how many queries are being sent to which database server and how much is cached. It also documents all the slow queries so we can evaluate and optimize them.

We're also using a few open source server monitoring tools and some home-crafted tools that send email alerts if it notices anything not working properly or if we're approaching our servers' limits.

Optimize code and then freeze it

Big spikes in traffic can bring to light all the bottlenecks and issues in your code. It’s better to try and address/eliminate them on your own terms rather than once all your clients are calling in a panic. We use tools like New Relic and ScaleArc to pinpoint the inefficient code.

Using a service like CloudFlare to add a level of caching to static files helps to reduce the requests that are sent to your servers. This means that you don’t need to serve up all the content all the time, and that your servers will have more resources available to handle the more complicated requests.

Don't Get Caught With Your App Down: Cloud Flare

Once you’re comfortable that your code is optimized and will be able to handle the load, avoid introducing any new features (and bugs) the week prior to Black Friday and Cyber Monday.

We generally have a "code freeze" on our production servers two weeks prior to Black Friday right through until just after Cyber Monday. The only thing that might possibly get released during this time is an emergency fix, but other than that we limit all other releases to reduce any risk of a bad release.

Expect the worst

This might sound bad, but at the end of the day the thing that can hurt you the most is expecting things to be okay. In fact, that's about the worst thing you can do!

We've outlined a few things here that we've done to prepare for Black Friday, but every app or custom development is different. Try to think through your app as best as possible and consider where anything could possibly cause a problem.

What will happen if you got 10x the traffic tomorrow?
What would happen if your app went down?
What would be the minimum downtime you could expect if your servers become overloaded?
Do you have a backup plan if that happens?

These are all questions you should ask.

This year, decide to do everything you possibly can to make it go smoothly. This year, don't let a lack of a decision be the thing that defines you.

You might also like: How to Build Your First Shopify App