In a hurry? Skip straight to the troubleshooting tips below. And remember: it’s all gonna be ok.
Something’s happened. You’re not sure what exactly, but you do know that metric is way too low to be normal. So how do you figure out what’s wrong? Is there even something wrong, or is it just a blip, seasonal slowdown, or something else entirely?
If the mere act of imagining this scenario gave you an uncomfortable knot in your stomach, welcome to the world of Product
Management Triage. This is when all your high-falutin’ strategy and touchy-feely customer development interviews come to a screeching halt, because Something. Is. Wrong.
Now, don’t get your hopes up too high; I don’t have all the answers – but most of the time, the blame can be pinned on a small number of usual suspects that are fairly easily identified and addressed. My hope is that referring to this troubleshooting guide at the first sign of trouble will help you quickly eliminate the most common problems that can suddenly crop up with a web product or business, and arm you with the right questions to ask of the right people in order to get to a resolution as efficiently as possible.
Before we begin – be prepared
If you’re going to lead your team out of the darkness, you need a few things in your toolkit – and if you don’t have them, you cannot effectively diagnose the problem, so you should always start here:
Open communication – Much like sobriety, you must take the crucial first step and admit there’s a problem. This is immediately followed by communicating with your team to maintain awareness of all the major changes happening to your website and app. If the signup page has been redesigned and signups suddenly fall, you can save hours of detective work by assuming there’s a correlation. Google Analytics’ annotations feature is a good place to start.
Analytics – You MUST be tracking as much as you can. If you’re not tracking something, how can you know you have a problem to begin with? I love Google Analytics for monitoring web traffic, Mixpanel for tracking user activity through our apps, and we use Dashing for a customizable real-time display of our key metrics – so we know there’s a problem as soon as (or shortly after) it manifests.
A friendly developer – If something’s broken, and you’re not a developer yourself, you need someone who can help you fix it. Without a machine-fluent buddy, you’re pretty much just a brain with no hands, which, um… ew. Negotiate hard with whomever you must, to clear their schedule and get dedicated developer time. Which brings me to…
Code-level access – For a lot of scenarios, you’ll need the ability to quickly deploy certain diagnostic tools in order to get a clearer picture of what’s happening. In lieu of this level of access, you could petition for a tag management system to get set up, or I could slyly recommend you check out the CodeDrop app on Filament, because it lets you drag and drop code snippets onto your site in seconds, and with zero hassle. But I would never be so direct…
These things are table-stakes for solving a serious problem – if you encounter resistance from the rest of your organization in acquiring these basic items, then by definition this is not being treated as A. Serious. Problem., and it’s up to you to either change that perception, or de-prioritize it accordingly.
The Usual Suspects
Okay, take a deep breath and resist that hot, prickly freakout sensation that’s crawling up from the bottom of your skull – even if everyone else is letting theirs run rampant. They need your calm leadership the most at this very moment.
- My traffic has dropped
- My signups have slowed down
- My activation rate has tanked
- Revenue is falling rapidly
- My churn has spiked
The “Duh” question(s)
- Is my traffic tracking correctly installed & configured?
- Has traffic really dropped, or is this a cyclical/seasonal decline?
Tips for Diagnosis
- Compare traffic for the day/week to the previous matching day/week. Expand to the previous 30 days to see how traffic has been trending. Then look at it this time last year. If there’s a significant drop from last week, and is counter to recent traffic trends, or even performance this time last year, then yes, there is a problem.
- If all traffic has completely dropped to zero…is your analytics tracking code still installed correctly on your website? (Seriously, you wouldn’t believe how easily this happens, especially with major redesigns going live).
- Look at your sources – which one(s) have fallen off the wagon, and by how much? I typically set the rough threshold at a ~20% drop – more than that, and something serious has happened with that source.
- Find relative changes in your individual traffic sources, using the Acquisition » All Traffic report in Google Analytics
- Similarly, check your referring domains, using the Acquisition » All Referrals report
- You dropped in search rankings – bigger drops can be associated with falling off the bottom of the search results page (going from page 1 to page 2 for a competitive keyword, for example)
- A valuable referring site no longer links to you, or the link has broken
- A featured article that links to you has moved into the archives, or was taken down
- The link for an effective paid ad placement is broken, or the ad spot has lapsed
- A seasonal decline, or public holiday in your biggest audience’s country/region
- If your biggest drop is associated with your “Direct / None” traffic source (as Google Analytics helpfully describes a huge chunk of your traffic), it’s ok to let out a few choice swear words at first, since this is a black-box category – since there’s no referring information, it’s impossible to know exactly where they came from. But all is not lost! With some more sifting, you can determine what their intentions might be, yielding clues as to their origin. Avinash Kaushik has a great post about embracing the Direct/None enigma (rescued from the jaws of obscurity thanks to Google’s Cache View).
- Now, don’t laugh, but…is there a new iPhone launch, or has some other major external event occurred? Seriously! Cast your mind back to June 25th, 2009 – the passing of Michael Jackson. That single event sent analytics folks scrambling for answers until the story broke wide.
- Google Analytics
- Free Rank Checker
- A good relationship with your top referrers – then you can just ask them what happened!
The “Duh” question(s)
- Is my signup flow functioning correctly?
- Are my signup Calls-to-Action’s (CTA’s) correctly linked?
Tips for diagnosis:
- Test the signup flow yourself (Pro-tip: if you use Gmail or have a Google Apps account, and need to quickly test signing up with multiple email addresses, just add “+[label]” and a number in front of the @ sign in your email address – for example [email protected] It’ll usually be recognized as a unique address by most web apps, but Gmail routes all messages to the original address – in this case [email protected]).
- Check your traffic (see above) – has there been a drop in the traffic source(s) that drives the most signups? (You can be forgiven if you don’t know which traffic sources those are. It’s one of the reasons we’re building Filament Insights – sign up for the beta!)
- What changes have been made to the signup page recently?
- Are your signup tracking events firing correctly?
- Look at the signup logs in your app’s database from the date of the drop to the present, and compare against the numbers shown in your analytics/dashboard. If there’s a discrepancy, there’s something wrong with the tracking mechanism.
- Did you recently redesign? You may want to check whether something in the new design is negatively impacting your users’ ability to sign up.
- Broken signup process
- Broken signup event tracking
- Screen resolution of your visitors not accommodating key signup CTA/design elements
- Browser compatibility issues interfering with important CTA elements
- Google Analytics annotations – Keep a running log of changes made to your marketing website
- Google Analytics browser resolution report – Check how your signup page appears to your most popular screen resolutions
- Optimizely – Easy A/B testing
- Screenfly – Check how your site appears at any resolution
- Qualaroo & Olark – Ask your users if there’s anything preventing them from signing up, or to contact you with problems
The “Duh” question:
- Is my onboarding process functioning correctly?
Tips for Diagnosis:
- Observe a sample set of volunteers as they walk through your onboarding process – bonus points if you can get them to narrate their thoughts as they proceed. Bet you dollars to donuts you’ll be surprised at some of the things that come up and conclusions they come to!
- Check the funnel conversion rates between each step in your onboarding process (i.e. all the steps that occur between signup and activation). If one of them has suddenly dipped, it’s usually a sign that something has happened there.
- Have you suddenly experienced a large increase of lower quality traffic? Sometimes even a single tweet from a major influencer can refer a huge number of visits to your site – but many of them will just be looky-loos who don’t convert into active users, or even venture beyond the signup page.
- Clarity is key in an onboarding scenario. Key text labels throughout the onboarding process may be missing/obscured in some way.
- Usertesting.com – For a few bucks, you can recruit volunteers to record videos of their experiences going through your website and onboarding process, and gain insights worth literally thousands of dollars worth of time.
- Mixpanel – Dive into funnel reports to see just where users are falling off as they go through your activation process.
- UserOnboard.com – Awesome teardowns of the best onboarding processes in the business. An amazing resource for getting inspiration as to things to test and optimize.
- Again, Olark and Qualaroo give your visitors a way to communicate any issues to you.
The “Duh” question:
- Is my checkout process working correctly?
Tips for diagnosis:
- Enlist a few people to run test purchases – try different product combinations, upgrade/downgrade scenarios, etc. Remember to use the “+label” Gmail trick!
- Check your purchase logs for a pattern – are you seeing lots of uses of a particular coupon code, sales from a particular country, or very similar names/email addresses associated with recent purchases?
- Check your sales funnel, to see where visitors are falling off.
- A rogue coupon code could have gotten loose. If you’re seeing a spike in use of a particular coupon code, google for it, and investigate results from RetailMeNot.com and other coupon aggregation sites. It’s possible someone shared it, and people are finding/redeeming it at a greater quantity than your marketing folks intended.
- Slow-loading checkout pages/confirmation pages breed instant frustration. There may be a script of some kind of slowing during the checkout process.
- Is there a problem with your SSL certificate? Sometimes, it can act up – a lot of the time, it’s due to there being an asset (image, CSS file, etc.) loaded insecurely on a secure page, which causes Chrome (among others) to freak out and warn the user that they’re in mortal danger and to run away, RUN AWAY!
- Did you raise prices recently? If so, you can expect that your average transaction value will have bumped up and sales volume will tank – but if the price is too high, you’ll be making less overall revenue.
- Olark – Get that sucker installed, and have your almost-customers tell you what’s wrong right from the point of frustration – even if it’s just to vent about the price.
- Usertesting.com – Watch videos of people trying to use your checkout process. Surprises galore await you!
Sounds like you should see a doctor! (Aw, c’mon a little levity in a fix-this-or-you’re-fired scenario never hurt, right?)
The “Duh” questions:
- Actually, there are no “Duh” questions here – only opportunities to learn, and make your product better.
Tips for Diagnosis:
- It’s simple, really: contact each of the folks who have churned out, and find out why they left.
- Confirm what was changed in the product just before Churn spiked – was there an interface update that’s proving unpopular? A key function or feature that’s now broken as a result of an update?
It could be anything, really, but typically if you have users who were previously paying for your product that suddenly decided to stop, it’s due to:
- a decline in functionality
- a lack of support, or
- an increase in competition
You know when a grizzled police chief credits “good old-fashioned police-work” for solving a difficult case? That’s what you have here. No tools will replace just diving in with your churned users and identifying the root cause of their sudden dissatisfaction with the product.
It really sucks when your metrics slide – but it sucks far worse to find out about it long after the fact. Good analytics coverage prevents such unwanted surprises, so keep calm and have your tools and resources ready.
“When you hear hooves, think horses – not zebras”
The simplest explanation is usually the right one. When you do have to dive in, look at the usual suspects first. Checking your entire funnel for the most common issues will most times lead to the quickest fix.
How are you supposed to keep an eye on all of this? It’s a great question – we’ve struggled with maintaining this visibility ourselves, which is why we’re building Filament Insights, a system that proactively notifies you of significant changes in the engagement of your websites. You should sign up for the beta!