Vanity metrics: good for feeling awesome, bad for action. (photo source: UK Guardian)
This is a guest post by serial entrepreneur Eric Ries. He was most recently co-founder and CTO of IMVU, which has more than 20 million registered users and generates $1,000,000+ in revenue per month. Eric is also a venture advisor to Kleiner Perkins.
How do you get to $1,000,000 per month in sales? By testing the right things. Eric is a metrics man.
Here is just one business-changing example, taken from the outstanding “How IMVU Learned its way to $10M a year” on Venture Hacks…
IMVU learned its way to product/market fit. They threw away their first product (40,000 lines of code that implemented an IM add-on) as they learned customers didn’t want it. They used customer development and agile software development to eventually discover customers who would pay for 3D animated chat software ($10M in revenue in 2007). IMVU learned to test their assumptions instead of executing them as if they were passed down from God.
Enter Eric Ries…
Vanity Metrics vs. Actionable Metrics
The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.
When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.
Now consider the case of an Actionable Metric. Imagine you add a new feature to your website, and you do it using an A/B split-test in which 50% of customers see the new feature and the other 50% don’t. A few days later, you take a look at the revenue you’ve earned from each set of customers, noticing that group B has 20% higher revenue per-customer. Think of all the decisions you can make: obviously, roll out the feature to 100% of your customers; continue to experiment with more features like this one; and realize that you’ve probably learned something that’s particular valuable to your customers.
Unfortunately, most analytics packages are configured by default to provide mostly reports on vanity metrics. That makes sense, since they are the easiest to measure and they tend to make you feel good about yourself.
For example, here’s a pattern I’ve witnessed in companies large and small. The company launches a new feature or new product, and a few days later, traffic (or revenue, or customers) starts going up. Everyone involved with that product celebrates. In fact, I’ve noticed that people tend to believe that whatever they were working on that preceded the metrics improvement probably caused the improvement itself. So the product guys think it’s the new feature, the sales guys think it’s that new promotion — I’ve even seen customer service reps be convinced it’s due to a new customer-friendly policy. In many cases the fluctuations are random or caused by unrelated external events. Unfortunately, the same mental trickery doesn’t apply when the numbers come back down. Human beings have an unfortunate bias to take credit for positive results and pass the blame for negative results.
Take the example of a product that has a weekly seasonality pattern. For products “on the Disneyland calendar” they will see higher usage on weekends and holidays. As a result, new initiatives that are launched on Thursday or Friday are likely to be judged a success when people come to work on Monday. Yet products unfortunate enough to be launched on Sunday may be judged a failure by Tuesday or Wednesday — unless the company is focused on Actionable Metrics.
There are some tips to getting to more actionable metrics:
1. Split-tests.
A/B experiments produce the most actionable of all metrics, because they explicitly refute or confirm a specific hypothesis. Either way, you can use split-tests to take action on anything from minor copy tweaks to major changes in the product or its positioning. However, not all split-tests are created equal. There is some value in the linear-optimization type tests that are a useful tactic in growing conversions. But the real value of split-tests comes when you integrate them into your decision loop: the process of putting your ideas in practice, seeing what happens, and learning for your next set of ideas. The tests that drive the most learning are the ones to focus on. A good rule of thumb is to ask yourself, “if this test turns out differently from how I expect, will that cast serious doubts on what I think I know about my customers?” If not, try something bigger.
Good third-party tools for A/B testing are hard to come by — most are too complex for most situations. If you don’t have an A/B system, you can use Google Website Optimizer or — if you have a software development team — build your own (for more implementation details, see “The one-line split-test, or how to A/B all the time” and “Getting started with split-testing“).
2. Per-customer metrics.
It’s important to remember, “Metrics are people, too.” Vanity metrics tend to take our attention away from this reality by focusing our attention on abstract groups and concepts. Instead, take a look at data that is happening on a per-customer or per-segment basis. For example, instead of looking at the total number of pageviews in a given month, consider looking at the number of pageviews per new and returning customer. Those metrics should be relatively constant — unless something interesting is happening with your product. So even a big rush of new customers shouldn’t change how many pages they each view on average, unless you’re getting a new kind of customer.
Similarly, if you’re increasing the engagement of customers with your product, that will tend to show up in the data for the returning customers. But if you just look at their aggregate data, you can miss important trends. I’ve often observed the following pattern: a big spike of customers joins thanks to a Digg or Slashdot mention. If a product has an average customer lifetime of two months, then after that period elapses, a huge number of customers can be expected to churn out all around the same time. But these effects are hard to keep track of, since customers are coming and going all the time. If you focus only on the number of pageviews, even if you limit it to returning customers, you might mistake a positive product change for something negative, because you launched it during a churn-dominated period.
Many analytics packages, including the much-maligned Google Analytics, have the ability to break down aggregates into per-customer or per-segment analyses. These can help make reports more actionable if you combine them with the Goal Tracking feature. For example, if you can tell which web referrers are driving the most traffic, that’s moderately useful. But if you can tell which are driving the most conversions, then you can start to make ROI-based decisions on where to invest your time in getting more traffic.
3. Funnel metrics and cohort analysis.
The best kind of per-customer metrics to use for ongoing decision making are cohort metrics. For example, consider an ecommerce product that has a couple of key customer lifecycle events: registering for the product, signing up for the free trial, using the product, and becoming a paying customer. We can create a simple report that shows these metrics for subsequent cohorts (groups) over time. Let’s say we create a weekly report. For each week, we then report on what percentage of customers who registered in that week subsequently went on to take each lifecycle action. If these numbers are holding steady from cohort to cohort, then we get clear feedback that nothing significant is changing. If one suddenly shifts up or down, we get a rapid signal to investigate.
The best thing about funnel metrics is that they allow you to boil down a large amount of information into a handful of numbers. If you don’t have the software to build these reports automatically, consider doing it by hand.
This is easy to do if the number of conversion events in relatively small — even if the number of customers is very large. For example, a typical website will have a 1% registration-to-purchase conversion rate. So even if you are registering 1000 new customers every day, those customers are going to result in something like 10 new purchases over their lifetime. So instead of getting fancy, use the good old index cards. At the end of each day, create an index card with that day’s date on it and the number of people who registered that day. Then, for each conversion that comes in, make a tally mark on the index card of the date that the person registered, not the date they purchased. For most products, this only requires you to maintain a week or two’s worth of index cards, since most products have customers that make purchase decisions relatively quickly. Then, on a weekly or monthly basis, gather up all the cards for a given cohort, and compute the conversion rate of the customers who registered in that period. That’s the number you want to focus on driving up.
4. Keyword (SEM/SEO) metrics.
SEM (Search Engine Marketing) and SEO (Search Engine Optimization) are great customer acquisition tactics, but they also can reveal important and actionable insights about customers, if we treat customers who were acquired with a given keyword as a segment and then track their metrics over time. For example, early on at IMVU we tried advertising for AdWords phrases that contained the name of a competitor’s product plus “chat.” We’d then take a look at key statistics for the cohort of customers that registered from each separate campaign. What we found were striking differences in signup and conversion rates depending on what competitor we brought the customer in from. That information is moderately useful in directing a marketing campaign. But it’s far more useful as an indicator of who the customer behind the numbers are. We eventually found that the highest conversion rates came from products that are primarily used by teenagers and young adults — a very different demographic than we thought we were serving. As a result, we started to adjust the mix of customers we were bringing in for usability tests, with dramatic results. For concrete examples of user feedback and testing, see the below video from an interview with Mixergy:
Here is a small sample transcript from the above video:
And so out of complete desperation, we were like, “Okay, fine, we’ll introduce a simple chat now feature.” It was a matching thing where you could push a button and you would be randomly matched with somebody else from around the world – the only thing you have in common is you both pushed that button at the same time.
And we did that, and all of a sudden people were like, “Oh, this is fun.” And then – then here’s what happened. So we bring them in and they do the Chat Now, maybe they meet somebody new who they thought was kind of cool. They’d be like, “Hey, that guy was neat, I want to add him to my Buddy List. Where’s my Buddy List?”
And we say, “Oh, no, no. You don’t want your own Buddy List. You want to use your regular AOL Buddy List” because that’s interoperability, network effects, all this nonsense.
And the customer’s looking at us like, “Well, that doesn’t make sense. What do you want me to do exactly?”
And we said, “Well, just give that stranger you just met your AIM Screen Name so you can put them on your Buddy List.”
And you can see the eyes go wide – they’re like “Are you kidding me?! A stranger on my AIM Buddy List?”
And we said, “But – but otherwise you’d have to download a whole new instant messaging client! And then you’d have to have your separate Buddy Lists.”
They’re looking at us like, “Do you have any idea how many instant messaging clients I already run?”
We said, “No, what, like two or three?”
And the teenager responds, “Duh! I run eight!”
They were already running, like, fifty clients! I mean, I had no idea how many instant messaging clients there were in the world. And we had this preconception like, “Oh, it’s a challenge to learn new software, and it’s tricky to move your friends over to the new Buddy List,” and all this other nonsense sitting in our heads that just, for our customers, looked at us like we were crazy.
Conclusion and Challenge
A common theme across all of these actionable metrics is the lack of really good action-oriented third party tools.
So I’d like to issue this challenge to all of you reading this post today: share your stories of actionable metrics and how you track them. If there are good tools that you have used, let us know. Most importantly, let us know how you customized off-the-shelf tools like Google Analytics to get more action-oriented. We’ll share the results in a future post. We’re looking for stories that embody these three principles:
1. Measure what matters. It’s tempting to think that, because some metrics is good, more metrics is better. That’s why vendors routinely list the thousands of reports they are capable of generating as a feature. The truth is, the key to actionable metrics is having as few as possible. Detailed reports are useful when we’ve diagnosed a problem and are looking for clues as to what’s gone wrong. But where does that diagnosis come from in the first place? Actionable metrics help us realize we have a problem and point us in the right direction to start solving it.
2. Metrics are people, too. Great metrics tools allow us to audit their accuracy by tracing reports back to the individual people who generated their data. This improves accuracy, but its more important effect is that it lets us use the same customers for in-depth qualitative research. Not sure what the numbers mean? Get the customers on the phone and ask them.
3. Measure the Macro. Lastly, even when we’re split testing the impact of a minor change, like a wording or a new button, it’s important not to get distracted by intermediate metrics like the click-through rate of the button itself. We don’t care about click-through rates, we only care about the customer behaviors that lead to something useful, whether purchase, retention for advertising CPM, or some other measurable “success” particular to your business model.
[From Tim: Here are a few options to get the juices flowing: The Better Google Analytics Firefox plug-in and six other tools for specific Google Analytics feature enhancement.]
###
Metrics are just one component of a new vision for entrepreneurship that I call the “lean startup”. You can learn more on the Startup Lessons Learned blog. For those that want to explore these concepts in comprehensive depth, including more real-world examples, there will be two all-day Lean Startup seminars sponsored by O’Reilly on May 29 and June 18 in San Francisco.
The Tim Ferriss Show is one of the most popular podcasts in the world with more than one billion downloads. It has been selected for "Best of Apple Podcasts" three times, it is often the #1 interview podcast across all of Apple Podcasts, and it's been ranked #1 out of 400,000+ podcasts on many occasions. To listen to any of the past episodes for free, check out this page.
Comment Rules: Remember what Fonzie was like? Cool. That’s how we’re gonna be — cool. Critical is fine, but if you’re rude, we’ll delete your stuff. Please do not put your URL in the comment text and please use your PERSONAL name or initials and not your business name, as the latter comes off like spam. Have fun and thanks for adding to the conversation! (Thanks to Brian Oberkirch for the inspiration.)
Thanks for this great comparison of actionable metrics vs. vanity metrics. As I push farther into the internet marketing world these things are of exceptional importance.
I particularly like the A/B split testing for finding actionable changes.
“Test their assumptions instead of executing them as if they were passed down from God”
Good advice. I find even ideas that ARE passed down from God need a little tweaking.
The reception from heaven isn’t always perfect…
Wow, Eric, thanks for some seriously thorough advice on a topic that’s very close to home for my product. As with all your articles, you’ve given the details to take action here…and really I have no excuse but to beef up the dashboard for my product now.
I particularly like your thoughts on “measure the macro”–because it is very tempting to measure the click, when that’s not necessarily the sale or the other conversion we’re most interested in. Wish I could attend your upcoming seminar!
Wow, this and a few other pieces I’ve read really get me motivated to delve into meaningful analytics. But then reality hits. When you actually try to start “doing” some of this stuff, it just seems like a 1000 piece puzzle.
I often ask myself if I’m better, at this stage in the game, to pump out tons of useful and interesting content instead of spending my time looking at numbers. I know it’s a bit of a catch 22, but that’s reality.
I guess I better get back and take another look at what I’m doing with analytics.
Cheers,
Adam
Interestingly, a lot of the vanity metrics are the focus of internet advertisers, and less so for ventures and entrepreneurs. For example, click-through rates are crucial to Adwords advertisers because it directly influence the cost and profitability of an ad campaign, and advertisers will relentlessly tweak something as simple as the Ad title to boost this click through number.
Eric Ries posts: “We don’t care about click-through rates, we only care about the customer behaviors that lead to something useful, whether purchase, retention for advertising CPM, or some other measurable “success” particular to your business model.”
Ad campaigns aside, this discussion touches on the core of what makes time spent on internet metrics worthwhile: the closer you are to direct customer feedback, in every way, the better the numbers that matter to YOU will look. Are your customers happy? Are they happy to hang out here with you on your site, in your business, doing what you want them to do?
Content, Content, Content. Tim talks about it all the time. Content is what drives readers.
I am really interested in what companies like http://www.kontagent.com are doing with measuring user engagement loops and virality. Especially as more companies adopt FB Connect and other social platforms as their viral distribution channels.
Digg’s first installment of Facebook Connect must be returning some very interesting data as every story published into the stream is tagged with several campaign tags according to link location within the story (I think they use Omniture)
The A/B test is incredibly valuable here and will give 3rd parties the data they need to fully leverage the power of Connect and measure the top responses from organic stream traffic.
Love the Zoolander shot!
Great post! I’m getting to be pretty maniacal about split testing on my landing pages these days — it seems to give the best return on effort… Its amazing to me how resistant people are to testing and tracking things that really matter — I guess people feel better when they can tell others that they had 100,000 hits on their website today and just ignore the fact that none of them made a purchase.
That was the most entertaining video chat transcript I’ve ever watched. Eric Ries covers a lot of ground, stays organized, and sounds truly passionate. And not a single “uh” or “um” to be heard.
The entire hour of the talk on mixergy doesn’t disappoint–check it out if you want more.
(P.S. Tim, you might want to study what it is he does so well for your “Random” chat with Kevin Rose.)
This is a great article and totally in line with my company’s (RJMetrics) way of looking at the world. We develop a tool that connects to our clients’ databases to perform cohort analysis, segmented customer lifetime value, etc. The most common question I get from potential customers is why would I use a tool like yours when I already monitor my page views and visitors in Google Analytics (or some other tool). I’ll probably just point them to this article in the future.
And Eric, if you’re looking at third party tools as you continue to do this kind of analysis, I’d love to talk.
I’ve been using Google Analytics on my websites and I’ve found it to be quite useful. I’m tracking what keywords are bringing in customers/potential customers and where they are going on my websites.
However, I still have a long way to go and this post has given me some important ideas.
I was expecting body fat % vs dead-lift. Profitability is also good through.
>For example, a typical website will have a 1% registration-to-purchase conversion rate.
This number is surprisingly uniform. It even applies to the average visit:sale ratio of downloadable software sites. See:
http://successfulsoftware.net/2009/04/23/the-truth-about-conversion-ratios-for-software/
Excellent post!
Glad my video helped. Eric Ries is doing amazing work.
As a huge fan of this blog, I was very excited when I saw this post today!
Analytics, Multivariate testing, and A/B split testing is what a huge chunk of my responsibilities are where I work.
Every day I am amazed at comparing what you think you know about your web visitors, and then what you learn from their actual actions and the inferred intent learned from analytics data, this combined with the outcomes of testing is usually very eye opening.
Use your analytics data to find opportunities where you can improve. Then use a/b testing or multivariate testing to improve those pages where those opportunties live (unless it’s simple enough to fix without having to test)
The best part is, it costs nothing (so no excuses!) to start learning and using:
Google Analytics – FREE
Google Website Optimizer – FREE
Your Ideas – FREE
Web Research if needed – FREE
Again – Great Post!
Wonderful write up on the perception of “what really works” as opposed to “what really looks good”. My hat goes off to Eric on this one.
Nice post, I have just started to use http://www.woopra.com as an analytics tool. It seems a bit easier to use against google analytics but I would opt to stay with the latter option for now. Also, SEO for firefox has interesting data that gives you an idea of how sites are ranking.
Thanks Tim & Eric.
Cheers
Jose
Great post Eric,
I am following your blog and as an entrepreneur it is constantly interesting to review what others have learned on the subject.
Peter Drucker stated “That which gets measured gets managed”. I mentioned this to family and friends regularly and show the point of following the right metrics using an example similar to below…
Say you want to improve your physical appearance, as you are looking a little overweight in your own opinion. You get on the scales and weigh yourself and then work for two weeks straight in the gym, running and eating well. When you jump back on the scales you notice not much has changed and become disappointed.
In reality there has been a change to your body. You are likely to have increased muscle and reduced fat, but will probably weigh the same. You will probably have more energy but lastly, you would have noticed visual results if you were measuring a different way. Essentially by measuring the size of your waist, arms, legs, etc. then you may have noticed more of a change.
The key is to measure what matters, not what is easy.
Great post. Me and my partner were talking about this same topic 2 days ago. Thanks for the informative post Tim and Eric.
@Rosa and @Blair: i agree with what you said. Well put.
@jose: I’m trying Woopra out too. I really dont like using GA that much.
Wow, thanks for the great comments! Keep the suggestions coming.
@Adam – analytics can indeed be daunting. My hope with a post like this is to give you a few concrete options you can get started with. If you’d like to talk specifically about how to get started, post another comment and I’m sure we could all brainstorm a few suggestions.
@Andy – great link. I can’t believe how often I see that magic 1% show up.
Thanks everyone for reading and sharing,
Eric
Testing and analyzing is something I need to do more of. Thanks for the this post. I’m going to check out a couple of links…
Jeff
Every few years in government, someone would decide that we needed to do some metrics. So, we did some metrics. Nothing came of it all except a waste of taxpayers’ money, but the someone was satisfied because he caused people to do some metrics.
So many people forget (or just can’t be bothered to do the necessary testing to success online. Thanks Tim and Erik for the wake-up call!
I’ve shared this article with my twitter followers because I think that it is very useful. The world of metrics is like a science by itself.
Great article!
Hey Eric – yeah, my question is similar to Adam’s –
When starting out and building traffic, and your monthly traffic is around 10,000, do you still think split testing is important?
How much can you just rely on learning and feel from the feedback you’re getting from your market?
At what point in size and sales does going by feel not make sense any more?
(I think I’ve got an idea of what you’re going to say, but I’m a little hard headed)
Personally, I’m partial to measuring and evaluating the following metrics:
Time on Site
Bounce Rate
Top Exit Pages
Conversion Rate: Immediate vs. Latent
New vs. Return Visitors
Number of products in cart
Abandons during checkout process
AOV
Navigational Behavior – Process flow & feature usage
One framework I like to use for metrics:
Metric Category: Example of metric
Outcome/Business driver: Leads, unique visits, revenue
Diagnostic: Conversion rate
Smoke Alarm: zero yield searches
Predictor/Leading Indicator: intent to purchase, net promoter score
Latent: correlations, voice of customer
Answers the following questions:
Outcome/Business driver: How well am I doing?
Diagnostic: How can I do better?
Smoke Alarm: What am I doing poorly?
Predictor/Leading Indicator: Will I do better?
Latent: Where is the opportunity?
A very effective post in helping marketers step back and pay attention the metrics that are actually impacting their organization. All of the sources of information and metrics available to marketers can easily cause analysis paralysis. I try to keep things in perspective with the following marketing analytics process… 1. Do something. 2. Measure something 3. Repeat!
Thanks for sharing.
Yes!!!! Fantastic post, Tim. And very timely too. Well done.
Great article. But still unsure how IMVU is making money?
I’m going to suggest that you could add a Theory of Constraints set of metrics to what you’ve come up with.
Throughput, inventory, operational expenses. At some point, you have to have product, that you move through your organization and it costs you something. This would be “per customer metrics” above, but is really at the core.
I would love to do some A/B testing on my site but I don’t like the idea of unique URLs in order to do it. For instance, the landing pages I would want to test have been optimized for search engine results and I don’t want to mess with that. Is there any way to do A/B testing with a single URL?
Thanks,
Tim
@Tim Chaves — It’s almost trivial to implement showing different content with the same URL using PHP. That is, technically speaking, your problem is easy to solve. I think some Google searching and reading would turn up an answer in no time.
Thanks to everyone who’s tweeting about this post – I’m @ericries on Twitter.
Eric
This is off topic but I was wondering if Brain quicken, which is now Body Quick contains any banned substances by the NFL. I like what I am reading about the success Olympic athletes have had using the product but i want to make sure its “safe” to take. By the way thanks for your book, time and constant blog that makes most of us expect more from our lives.
-Jeff
@JC,
Thanks for the comment, Jeff. To my knowledge, the product has no banned substances for any professional organization, whether NFL, IOC, MBA, NHL, etc. There have never been any false-positives in the 8 years that the product has been on the market, and it’s been used in all of the aforementioned leagues.
If you are a competitive athlete, be sure to keep an eye on a HUGE announcement on this blog in the next week or so!
All the best,
Tim
@Tim Chaves – You said:
“the landing pages I would want to test have been optimized for search engine results and I don’t want to mess with that. Is there any way to do A/B testing with a single URL?”
You dont have to worry about that while the test is going on if you’re using Google website Optimizer – from their FAQ directly.
“Website Optimizer is designed to keep your original content visible in the HTML source code of your page at all times. As a result, your original content is visible to crawlers, which means there should be no major impact on search engine ranking. However, if you implement changes to your content after using Website Optimizer, they’ll have the same effects as any content changes that you would typically make to your website.” (http://www.google.com/support/websiteoptimizer/bin/answer.py?hl=en&answer=63382)
If you’re worried about it after the test over the long run, it would be in most cases better to have a better performing page with lesser traffic and then go after building back up the traffic that you “might” lose from any changes, than keep with a high-traffic, low-performing page.
Test to get the best performing page and tweak it for SEO . A better performing page should help you to get better rankings anyways over the long haul.
Great article! This is top-notch writing. Your blog is really great! Thank you very much for the informative and creative content!
> When starting out and building traffic, and your monthly traffic is around
> 10,000, do you still think split testing is important?
That is more than enough traffic to split-test against. In fact, when your traffic is small is the best time to do dramatic tests, because if you really screw things up the consequences are much lower.
> How much can you just rely on learning and feel from the feedback you’re
> getting from your market?
How can you know the feedback you’re getting is really representitive and not just from your noisiest customers?
> A very effective post in helping marketers step back
> and pay attention the metrics that are actually
> impacting their organization. All of the sources of
> information and metrics available to marketers
> can easily cause analysis paralysis. I try to keep
> things in perspective with the following marketing
> analytics process…
Thanks, glad you liked it. I would add that this is not just for marketing – the essence of this approach is to integrate actionable metrics into all aspects of your business operations. THis is especially true in product development – otherwise, how do you know you’re making your product better and not worse?
> 1. Do something. 2. Measure something 3. Repeat!
This is the fundamental feedback loop that powers all startups – for example, a software startup is just a big catalyst that transforms Ideas into Code. Everything else is a side-effect. So the fundamental feedback loop is
Ideas (Build) Code (Measure) Data (Learn) Ideas (repeat)
Tim Robinson has a quality, low-price split-testing software called Conversion Chicken: http://www.conversionchicken.com You can get a free-trial of it. The software in my opinion is better than Google Analytics. Anyone who uses GA makes there testing a slow, painful process.
Awesome post Eric.
Actually there is one point that gets skipped at the beginning, which is the initial website design, obviously before anything exists, nothing can be measured or optimised.
We can all notice that there is a tendency to have certain elements on certain areas of the page of the majority of websites (for example logo / name on the top left), which I guess depends on the fact that we are accustomed to naturally focus on certain areas in a habitual way, but has this been actually validated?
Can you recommend any resource / tip to get a design that will right at the beginning give a sense of familiarity to the user? Something to avoid instead? Top tips / top mistakes to avoid when doing the initial design?
Tim,
in your first video with Kevin you mention page redesign to improve conversion for one of your investee, so maybe you can recommend something too?
Many many thanks, this blog gets getting better and better and I cannot recommend it highly enough.
Dan
PS: Tim, you love Malbec, have you tried Grignolino or Barbaresco from the Piemonte region in Italy? I think you will enjoy them.
Hi Dan,
I hope to have some specific examples of design on the blog soon. Keep an eye out, and thanks for the wine tips!
Tim
Tim,
Great site, great book.
I really enjoy when you write. I also enjoy when you allow others (such as this wonderful post and others such as Pavel’s), but I have one suggestion:
When you allow guest authors to write you should put a different color for their responses to comments.
When you write a blog I usually just scroll down and look for your green colored comments -an easy thing to spot. I think it’s a very useful thing when you write a blog, and it would be equally useful if you were to give the guest writer a (for example) red colored comment. This would help the readers gain more knowledge from these experts.
Thanks a lot.
P.S. I think you’ve gotten enough comments about your book on the Ivy League that it warrants an article. I know I’d be interested.
I need to find actionable metrics in nearly all areas of my life!
great post! I notice very often online stores get too hung up on finding out what happened instead of focussing on why.
[Moderator: Name Removed.] We develop a tool to help them with just this. We focus on the merchandise in each store and then connect the metrics on these to marketing strategy, merchandising, category mgmt and so on. Stores can use this say to find not just that shoppers from Facebook aren’t converting but why that’s the case and which of their products are a better fit for that demographic.
Eric if you are still evaluating third party analytical tools, we’d be more than happy to chat or set up an exclusive demo account.
Sorry Scott, that wasn’t intended as a reply to you.
> Actually there is one point that gets skipped
> at the beginning, which is the initial website
> design, obviously before anything exists,
> nothing can be measured or optimised.
Thanks for raising this issue – it gets right to the heart of one of the most challenging questions for new products: What is the minimum viable product? As soon as humanly possible, you want to be in a world where you can test your assumptions rapidly. As soon as you have something in customers hands, you are living in that world, even if the feedback is really negative. From then on, every feature suggestion/bugfix/change is automatically split-testable. You always have the old “bad” version to test against.
For more on the MVP, take a look at this interview:
http://venturehacks.com/articles/minimum-viable-product
Thanks for stopping by!
Eric
Marc and Josh,
That is very helpful, I really appreciate the input.
Tim
Thanks for a great posting, this is near and dear to me and really speaks to the core of the problem. This is far reaching and certainly prevalent in every size and shape organization. I have been preaching actionable metrics that matter for years, but never as clearly as stated here.
Aloha Tim, I was sent this new website link for computational mathematics..what’s you’re take on it?
http://www.wolframalpha.com/screencast/introducingwolframalpha.html
Hola Tim,
Greetings from Beijing!
Intense loneliness? Enjoy a tea ceremony and some good company.
Hope all is well.
Thanks for the feedback Eric – got the picture – beginning an initiative immediately –
Very well-written post.
Very informative post.
Nice blog by the way, I love the design.
Minah
Thanks for this great post. I agree content is key!
Regards
Marieme
Some of you might notice, that — by request — Eric’s replies are now also highlighted. Thanks to Matt Mullenweg (http://ma.tt) for hacking the PHP!
All the best,
Tim
Thanks for the great tips Eric,
My question(s) to you are the following:
* Which is the most effective way to do split testing if you little/no background in creating websites?
* Would you recommend split testing also on Google Adwords?
* Lastly, what 3 metrics would you consider the most essential overall for an ecommerce based biz? or a service based biz with an online presence?
Thanks for taking the time to read this.
Best
Jose Castro-Frenzel
What a fantastic post. Thank you, Eric, for sharing tips that go beyond simply analysing a server log.
Tim, one of the things I continue to enjoy about your blog – week after week – is how much I continue to learn. It is like a weekly hit for my addiction. Vielen Dank!
Fresh insights — thx Tim & Eric (not to be confused with the Awesome Show makers, altho they’d make great guest post contributors).
I enjoyed the video clip, who transcribed? You’re a fast speaker, Eric, and very lively/fun to watch.
Hi Tim,
You inspired me to write a book. I did. It’s now published, selling, etc.
Where can I send you a signed copy?
With warmest regards,
Sunday
Hello there Tim!
I really like your stuff but living in Brazil I sometimes find it difficult to apply it to my reality. I saw you once say that what one should do to perfect a task is to understand the rules, find the loopholes, and also find and focus at what you’re good at it. I can relate to all of that since I have lived for many years both in the States and in Europe but how do you do it when the rules are not clear? I say that because in Brazil that has been the problem for me. I work in a sector where nothing is clear and nothing is really out there, people keep secrets and the only way to know things is to be an insider. I am young and have just started my career and for the moment I am an outsider.
Do you see my drama? Do you have any words you can share?
Thanks a lot!
D.
Tim
I just got back from my first mini retirement. I spent three weeks in germany. I took some of the 4 hour work week ideas to make it happen. The best part was doing it while between jobs. Because of your book I took a step and grew as a person.
Your right reality can be bent.
Thanks again
Leonard
Right on the mark, guys.
I have a very successful client that does millions in rev. each year. They have dozens, if not 100’s of sales pages up – and they’re just now starting to test!!
With their success, I’d figured they were all over it.
Amazing how overlooked this essential component can be.
And how golden it is when implemented correctly.
Tim,
Love the book, addicted to the blog. Might I request a little more emphasis on “muse” ideas for blog posts. I feel like a great deal of your lifestyle design depends on the muse and yet it doesn’t receive too much attention on the blog. Keep the posts coming!
Thanks for your time
Ben
Tim,
I’m an example of the type of person you were speaking to in your book…investment banker right out of school, working 80+ hours a week – while hating my job, etc. I’d like to own my own business as soon as possible and am willing to leave my job tomorrow. On page 60 of your book you say “find someone who’s done it and ask for advice on how to do the same”. I’d love to quit my job and follow you around for a year picking up as much knowledge as possible, while helping in anyway I can. What do you think?
Taylor
Awesome stuff, guys. I’m a big fan of testing!
Hey Tim,
I hope the writing process on your next book on Metrics is going well. You seem indeed to be taking action already in that regard. I just watched yet another TED video (…) and you may wanna watch “Hans Rosling shows the best stats you’ve ever seen”.
You seem indeed to be putting a lot of effort on collecting the right data and making it speak properly. This might give you insight on expressing it in really cool and intelligible ways.
I believe strongly that data (or say metrics) VISUALISATION is also a tremendously KEY point in the CYCLE involving data input, actionnable metrics extraction, and decision/strategy. The keynote speaker mentions in the end the tools that he used. Jonathan Harris, in “Jonathan Harris : the Web’s secret stories” also uses fantastic tools to make the data speak. All this seems to go in the fasion of intelligible interfaces.
Since you are interested in data, you certainly have looked into the huge efforts that are being done on semantic data. Metrics are undoubtedly gonna take an IMMENSE importance when the Semantic web becomes true. All that available intelligible data will litterally crave for metrics to make them tell a story ! I can’t wait for those times !
Thanks for the book and I hope to read you again soon,
Warmly,
Youcef.
Great article in NYTimes today and that speaks to Ad Agecies hiring individuals strong in data analysis and how Mad Men have changed :
Put Ad on Web. Count Clicks. Revise. http://is.gd/LHSK
@Jose –
* Which is the most effective way to do split testing if you little/no background in creating websites?
Start with third-party tools like Google Website Optimizer. If anyone else knows of good general-purpose split-testing tools, please share.
* Would you recommend split testing also on Google Adwords?
Yes, especially on these. Tim has already written about how to do split-testing of ad copy (by turning off Google’s auto-optimization feature). I would also split-test landing pages and, most importantly, keyword-landing page fit.
* Lastly, what 3 metrics would you consider the most essential overall for an ecommerce based biz? or a service based biz with an online presence?
1. Cost to acquire a new customer
2. Liftetime value each customer acquired
3. Viral or retention rate of each customer (depending on which fundamental driver you use): http://startuplessonslearned.blogspot.com/2008/09/three-drivers-of-growth-for-your.html
Brilliant stuff….already see 3 things I need to change.
This is great stuff, I had to blog about it. too bad I came late to the party – I need to pay more attention to this blog!
[…] According to a recent study, Marketers are more likely to “monitor” than act on or react to Internet data. While 79% of businesses reported capturing Internet traffic information, only 30% of them actually modified their Websites as a result of traffic analysis. It makes me wonder if this move to be accountable by marketers is only important to them when they are measuring vanity metrics versus actionable metrics. […]
[…] swimming in a sea of data about our products, companies, and teams. Too much of this data is non-actionable. That’s because many of our… carry on reading. Share and […]
Hello all –
I completely agree with all the positive feedback on Tim’s initial post, enough so that we’re going to write a story about companies that are embracing the strategies Tim and Eric champion like Actionable Metrics and Minimum Viable Product for an upcoming issue of Inc. magazine. Our best stories are case studies that share not only the good things folks are doing, but also some of the mistakes they have learned from. If you have a good story to share, or know someone who does, drop me a line at darren_dahl@yahoo.com.
Now I need to figure out how to split test homepages using wordpress.
Vanity metrics are used for KPI alarms.
Not for driving but securing your business.
Wow! This is a great post. So many people I work with are either novices when it comes to analytics or masters. The key is helping those who have little experience with metrics getting comfortable using and understanding them.
I think that your recommendation for A/B testing is spot on for two reasons. The first is that A/B testing is easy to do (Google website optimizer). The second is that it gives you valuable data that can dramatically improve your conversion rates. Usually that’s all it takes to understand the value of analytics and more importantly get hooked on using them!
Thanks for the great blog Tim!
[…] 17. Many times, what we don’t know we don’t know is what we need to know. The first step is figuring that out. Eric Reis lean start up http://www.fourhourworkweek.com/blog/2009/05/19/vanity-metrics-vs-actionable-metrics/ […]
Hi Tim, after months in which I got up in the morning with no energy and have no energy and passion in life, I have decided to leave my envied marketing role in a multinational to start a PhD at 27 year old, going down to 25% of my current salary. Your book gave me part of the strength to make this decision, giving me the self confidence that by asking the right questions I can change my life. My next power question would be: how can I do what I like (PhD) and learn as much as doing what I did not like anymore (marketing job). Thanks for your inspiration! I will be in Geneva until Sep, doing a mini-retirement in California for 2 weeks in Aug, and then move to Paris.Any chance we can meet for a quick cha in any of these places?Ciao! Marco
[…] tech players including investor Josh Kopelman and entrepreneur Eric Ries have heralded cohort analysis as a preferred analysis technique. Cohort Analysis is heavily […]
[…] actionable metrics for testing. A lot of metrics are just bullshit. They aren’t actionable; you can’t do anything with them. AARRR […]
[…] data and metrics. You need results that you can actually do something with. There have been some great posts on actionable metrics. I encourage you to read them […]
Thanks for this article! As a web designer there are some golden gems as to how to design and test a website’s effectiveness.
[…] on the subject come from Dave McClure (“Startup Metrics for Pirates”) and Eric Ries (“Vanity Metrics vs. Actionable Metrics”). A good summary of these ideas, wrapped around a case study, can be found on the Ben […]
[…] overload at work to find the “actionable” information and not to get lost on the “vanity” […]
[…] Iterations hinge on the fact that you first measure things that matter (see the great post on vanity metrics, also by Eric) and then extract knowledge that feeds into your next cycle through the loop. Subvert […]
I realize that understanding the vanity metrics and actionable metrics would change the way I did business entirely, and increase my productivity. actionable metrics are something we can measure what we can then act on.vanity metric is one that we can ‘t really change, and therefore can not test. The key to all this is deciding which are actionable and which are vanity and then diverted out two.
Great article…thanks for opening my eyes even further to the importance of metrics. I am also one of those who have blindly thrown up content and hoped for the best finding that my expectations were not met. Testing will be a big part of my IM future! Cheers!
Thanks for your write-up. One other thing is when you are disposing your property yourself, one of the difficulties you need to be aware of upfront is just how to deal with property inspection reports. As a FSBO retailer, the key about successfully moving your property and saving money upon real estate agent income is know-how. The more you know, the smoother your sales effort will probably be. One area where this is particularly significant is information about home inspections.
hello ilike tihs article. Awesome stuff, guys. I’m a big fan of testing!
Four Hour Work Week changed my life.
Thanks for the post on metrics – it’s always a good reminder.
Great advice and Information. I must revise what we are currently measuring and question what really matters to our business.
I like the article. I found it a bit late.
Has the follow up with the testing tools from the challenge been written or updated?
Joe
http://www.appsee.com/ have an interesting system in place to dynamically track the user’s experience in your mobile app in addition to the traditional metrics. The fact that they don’t require you to pre-define what you are measuring allows for a quick re-evaluation method to look at prior performance, while their session recording lets you see firsthand what isn’t working in your app.
Overall, they are one of my favorites in the analytics space.
Is your site restricted to specific countries?
Microsoft Dynamics CRM plus ClickDimensions is a great tool for connecting all marketing efforts (traditional and digital) to Leads, Qualified Leads and then ultimately sales. A tool like ClickDimensions coupled with the already robust Dynamics CRM is one of the best tools in the industry because digital/email analytics and the CRM are all in one system. So you’re not passing information between systems like with other solutions which can lead to limited and disparate metrics. Having a central repository of real-time metrics is key for driving meaningful change across an organization.
Awesome article. Thank you!
Love this stuff
Yeah. It was a great article. But I had dilemma over my head that how can a company measure its business performance which has not gone online?
I am just wondering how can we apply the same principles to offline business ….