Overview
It is understandable that, as companies grow, they become more risk
averse. When you're small there really isn't much to lose, but as you
get to scale, there is quite a lot on the line, and many people from
across the company are there to try to protect those assets.


[Process @ Scale]


The modern alternative to roadmaps discussed here addresses both of
these concerns. Teams work
on the prioritized business objectives determined by the leaders; we
share our key results transparently; and we commit to high-integrity
commitments when critical delivery dates are needed.


[They want some visibility into what you are working on and
assurance that you are working on the most important
items.]
[They want to be able to plan the business and need to know when
critical things will happen.]


If the feature you're working on is to add PayPal as a payment method,
and the reason is to increase conversion, then be sure to always show
the current conversion rate and the result you're hoping to achieve.
Most important, after the feature goes live, be sure to highlight the
impact on that conversion rate.
If the impact was good, celebrate it. If the impact was not what was
hoped, then emphasize to everyone that, while you did ship the feature,
the result was not successful. Point out specifically what was learned,
but explain that you have other ideas for ways to get the desired
result.
The goal is that over time (it can take as long as a year), the
organization moves its focus from specific features launching on
specific dates to business results.
For this to work, it's important to acknowledge the two big reasons why
stakeholders especially are so attracted to roadmaps:


The goal is that over time, the organization moves its focus from
specific features launching on specific dates to business results.


Many product teams want to move off product roadmaps, but their
organizations are old school, addicted to the outdated quarterly product
roadmap. As a result, they don't see how to transition their
organizations forward.
Here's what I advocate in this case: Plan to continue with your
existing roadmap process for six to 12 months. However, starting
immediately, every time you reference a product roadmap item, or discuss
it in a presentation or meeting, be sure to include a reminder of the
actual business outcome that feature is intended to help.


[CHAPTER 60][Weaning an Organization Off Roadmaps]


Your specific success measures will depend on your goals, but
ultimately, you're looking to compare the team's effectiveness in
delivering business outcomes; that is, how well do the pilot teams
accomplish their objectives versus the others or compared to how they
did in the past?
Given the nature of the experiment, your comparisons will be
qualitative, but that doesn't make them any less compelling.
If things go well, you'll likely end up with several other teams eager to adopt. If things
don't go well, you might decide this technique is not a fit for you, or
you might decide to make adjustments.
To maximize the chance of the pilot teams having a good outcome, we
carefully consider the people involved, their location, and their degree
of autonomy. Ideally, we have people who are open to new ways of
working, the key people on the team are co-located, and the team is
largely in control of how they work and not so dependent on other teams
that still work in the old way.


Some people in your organization love change, some want to see someone
else use it successfully first, some need more time to digest changes,
and a few hate change and will only change if they're forced to do it.


Earlier in this book, we discussed the technology adoption curve and how
this theory describes how different people will accept change. It turns
out this theory also applies to our own organization and especially to
how we make changes to how that organization works.
Some people in your organization love change, some want to see someone
else use it successfully first, some need more time to digest changes,
and a few hate change and will only change if they're forced to do it.
If you get too excited and roll out a significant change to everyone in
the organization at once, then the laggards (those that hate change) may
resist or even sabotage your efforts.
Rather than fight this reality, we can embrace it. One of the simplest
techniques for facilitating moving to new ways of working is the use of
pilot teams. Pilot teams allow the roll out of change to a limited
part of the organization before implementing it more broadly. The idea
is that you look for a product team to volunteer to try out some new
techniques. You let them run for a while (usually a quarter or two) with
this new way of working and see how this goes.


[CHAPTER 59][Pilot Team Technique]


This sounds straightforward enough, but many problems arise because the
vast majority of these Agile coaches don't have experience with
tech-product companies, so their experience is limited to delivery.
Therefore, they would more accurately be considered Agile delivery
coaches. They understand the engineering and release side of things, but
not the discovery side of things.
So many product companies have experienced this issue that it created
the need for coaches that do have deep experience with product companies
and the key product roles, especially product management and product
design. These individuals are often called discovery coaches.
Discovery coaches are typically former product managers or product designers (or former
leaders of these areas) and they have usually worked for, or with,
leading product companies. So, they are able to work side by side with
actual product managers and designers---not just reciting Agile
platitudes, but showing the team how to work effectively.
Every discovery coach has his or her preferred way of engaging with a
team, but they are usually engaged with one or a small number of product
teams for a week or so. During this time, they help you through one or
more discovery cycles of ideation; creating prototypes; and validating
the prototypes with customers to gauge their reactions, with engineers
to evaluate feasibility, and with business stakeholders to assess
whether this solution would work for your business.
It's hard for me to imagine an effective discovery coach who doesn't
have first-hand experience as a product manager or product designer at a
modern product company. That's likely one of the main reasons there's
a shortage of discovery coaches today. It's also important that the
discovery coach understand how to include engineering in the mix---being
sensitive to their time, but understanding the essential role they play
in innovation.
Discovery coaches are not unlike Lean Startup coaches. The main
difference is that Lean Startup coaches often focus on helping a team
discover not only their product, but also their business model, and
their sales and marketing strategy. Once the new business has some
traction, the discovery is usually more about continuously improving an
existing product in substantial ways rather than creating an all-new
business. Because of this difference, many Lean Startup coaches don't
have the necessary product experience. My view is that product discovery
is the most important competency of a new startup, so I believe an
effective Lean Startup coach must also be very strong at product.


Discovery coaches are typically former product managers or product
designers and they have usually worked for, or with, leading product
companies.


Discovery Coaches
As teams moved to Agile methods (they usually start with Scrum), many
companies decided to contract with or hire an Agile coach. These coaches
help the broader team---especially engineers, QA, product managers, and
product designers---learn the methods and mindset involved in moving to
Agile.


I first met the Google Ventures (GV) team many years ago when they were
just getting started. They are part of Google's investment arm, but
even more valuable to the startup than their money, GV created a small
team to go in and help the companies they invest in get their product
efforts off to a good start. Their model is to typically spend a week
with the startup---rolling up their sleeves and showing them how to do
product discovery by doing it with them side by side.
I also know several other
proven product people, known as discovery coaches, who do essentially
the same thing for the teams they are helping.
In any case, during this week of intense discovery work, you and your
team will likely explore dozens of different product ideas and
approaches, with the goal of solving some significant business problem.
You'll always end your week by validating your potential solution with
real users and customers. And, in my experience, the result is
consistently big learning and insights---the kind of learning that can
change the course of your product or your company.
Within this general framework, discovery coaches advocate a variety of
different methods to help the team though the process and get big
learning in just five days.
After working with more than 100 product teams, and refining their
methods as they learned what worked well and what didn't, the GV team
decided to share their learnings in a book. The book is titled, Sprint:
How to Solve Big Problems and Test New Ideas in Just Five Days, by Jake
Knapp, John Zeratsky, and Braden Kowitz.
The authors lay out a five-day week that starts with framing the problem
by mapping the problem space, picking the problem to be solved and the
target customer, and then progresses into pursuing several different
approaches to the solution. The team next narrows down and fleshes out
the different potential solutions, then creates a high-fidelity user
prototype---finally, putting that prototype in front of actual target
users and observing their reactions.
And, yes, you can absolutely do this all in a week.
Sprint spells out the authors' favorite techniques to accomplish each
of these steps, and if you've read this far, you'll recognize all of
them. But what I like so much about the GV book is that, when teams are
getting started, they often crave the structure of a proven,
step-by-step recipe. The book spells out exactly this over nearly 300
pages, with dozens of examples from great products and teams you'll
recognize.
There are several situations where I recommend a discovery sprint, starting with when
the team has something big and critically important and/or difficult to
tackle. Another situation where a discovery sprint helps is when the
team is just learning how to do product discovery. And yet another is
when things are just moving too slow and the team needs to recalibrate
on just how fast they can and should be moving.
Sprint is another must read book for product managers, and I highly
recommend it.


A discovery sprint is a one-week time box of product discovery work,
designed to tackle a substantial problem or risk your product team is
facing.


I find that many teams, especially those new to modern product
techniques, are looking for a structured introduction to modern product
discovery. In this chapter, I describe the concept of a discovery
sprint.
A discovery sprint is a one-week time box of product discovery work,
designed to tackle a substantial problem or risk your product team is
facing.
A discovery sprint is definitely useful for more than just
transformation. It could just as easily be considered a discovery
planning technique or a discovery prototyping technique. But I find
it's most helpful in bringing all these things together, so I choose to
include it here.
Some people use the term design sprint rather than discovery sprint,
but as the purpose of the work---when done well---goes significantly
beyond design, I prefer the more general term.
Further, if your company has been struggling with the concept of MVP,
this is a very good way to start getting the value from this key
technique.


[CHAPTER 58][Discovery Sprint Technique]


Overview
So far, we've been discussing techniques for discovering successful
products. But it's important to acknowledge that getting product teams
and companies to apply the new techniques and work differently is often
easier said than done.
Partly this is because people are people. But mainly it's hard because
the changes are so often cultural.
As a very explicit example, moving from mercenary-style, product
roadmap-driven, output-focused teams, to truly empowered, accountable
product teams that are measured by business results, represents a major
cultural shift and a substantial handoff of power and control from
management to the individuals on the teams.
Believe me, this is not the type of change that happens easily.
Fortunately, there are techniques that can help the organization through
this.


[Transformation Techniques]


Even worse, DVD sales were starting to lag, and a Hollywood backlash
further muddied the situation. Then there were challenges with
fulfillment logistics, difficulty maintaining DVD quality, and trying to
figure out how to do all this in a way that covered costs and generated
some cash.
Kate Arnold was the product manager for this small team, and the team
knew they needed to do something different.
One of many tests they tried was to move to a subscription service. The
idea was to get people to sign up for a month, and offer them unlimited
movies. Would that be perceived as better enough to get them to change
their media consumption behavior?
The good news was that, yes, this approach really did appeal to
people. A flat monthly fee and all the videos they could consume sounded
pretty great.
The bad news is that the team created some real problems for
themselves. No surprise that Netflix customers wanted to rent mostly
newly released feature films; yet, these were much more expensive for
Netflix to stock, and they would need to stock so many copies of these
that they'd very likely run out of money fast.
So, the product challenge became how they were going to make sure Netflix customers could
watch a set of movies they would love, yet wouldn't bankrupt the
company.
They knew they needed to somehow get customers to want and ask for a
blend of expensive and less expensive titles. Necessity being the mother
of invention, this is where Netflix's queue, ratings system, and
recommendation engine all came from. Those were the technology-powered
innovations that enabled the new, much more desirable business model.
So, the team got to work. In three months' time, the team redesigned
the site---introducing the queue, the rating system, and the
recommendations engine all in support of Netflix as a subscription
service.
They also rewrote the billing system to handle the monthly subscription
model (a funny little side story is that they launched without this, as
they had the 30-day free trial month, which bought them the extra time
they needed).
With so many moving pieces and interconnected efforts, the daily
standups included just about every person in the company.
Between working with the co-founders on the strategy, validating
concepts with the users, assessing the analytics, driving features and
functionality with the team---and working with finance on the new
business model, marketing on acquisition, and the warehouse on
fulfillment---you can imagine the workload Kate faced on a daily basis.
Yet, the team got the new service up and running and used this to power
and grow their business for another seven years, until they disrupted
themselves again by moving aggressively to the streaming model.
Kate would be the first to credit a pretty amazing team, including some
exceptional engineers, and the vision and courage of the founders. But I
would argue that without Kate driving for the technology-based solutions
that could power this business, there's a good chance Netflix as we
know it never would have happened.
One more interesting little aside about early Netflix---when they were struggling for
cash early on, they offered to sell themselves to Blockbuster for $50
million, and Blockbuster turned them down. Today, Blockbuster is in the
dead pool, and Netflix is worth more than $40 billion.
Kate is now a product leader in New York City.


Those were the technology-powered innovations that enabled the new,
much more desirable business model.


Netflix is one of my all-time favorite products and companies. But back
in 1999, a then very young Netflix---based in Los Gatos, with fewer than
20 employees---was on the edge of going bust. They had a couple of
experienced co-founders, including the now legendary Reed Hastings, but
the problem was that they were stuck at about 300,000 customers.
They were essentially providing the same general pay-per-rental
experience that Blockbuster provided, just an online version. There
were, as always, some early adopters, and some people lived in places
that didn't have a video store, but in truth there wasn't much of a
reason to rent DVDs via the U.S. Postal Service when you could just stop
by the local Blockbuster store on the way home from work. People would
rent once from Netflix and then quickly forget about the service. They
didn't seem very willing to change. The team knew that the service
wasn't better enough to get people to change.


[CHAPTER 57][Profile: Kate Arnold of Netflix]


A walkthrough is when you show your prototype to a stakeholder and you
want to make sure they see and note absolutely everything that might be
a concern. The purpose is to give the stakeholder every opportunity to
spot a problem. The product manager usually drives, but if the
stakeholder would like to play with the prototype we are happy to let them. You are not trying
to sell them anything, you're not trying to test on them, and you're
definitely not trying to hide anything from them.
I have seen many inexperienced product managers do a walkthrough with a
prospective customer when they should have prepared a product demo. Or
another really common rookie mistake is to do a product demo during a
user test, and then ask the user what they think.
Be sure to be clear about whether you're doing a user test, a product
demo, or a walkthrough. And, be sure you're skilled in how to do all
three.


There are three very different techniques for showing the prototype,
and you have to be careful to use the right technique for the right
situation.


User Test versus Product Demo versus Walkthrough
Throughout this book, I have talked about "showing the prototype." In
truth, there are three very different techniques for showing the
prototype, and you have to be careful to use the right technique for the
right situation.
A user test is when we test our product ideas on real users and
customers. It is a qualitative usability and value-testing technique,
and we let the user drive. The purpose is to test the usability and
value of the prototype or product.
A product demo is when you sell your product to prospective users
and customers, or evangelize your product across your company. This is a
sales or persuasive tool. Product marketing usually creates a carefully
scripted product demo, but the product manager will occasionally be
asked to give the product demo---especially with high-value customers or
execs. In this case, the product manager does the driving. The purpose
is to show off the value of the prototype or product.


You need to help protect your company's revenue, reputation, employees,
and customers you've worked so hard to earn.
In this chapter, I name the main stakeholders in a tech-powered product
company, discuss their typical concerns and constraints, and explain how
the product manager would test business viability with each area.
While this is a very common list, and most or all of these areas
probably apply to your products, it is also very common that any company
will have one or more special stakeholders that are unique to the
business. Just because it's not listed below does not mean it's not
absolutely critical for you to deal with.
The last thing you want to have happen is that your team moves forward
and takes the time to commercialize the solution and deliver a shippable
product, only to find  out
that you can't ship because you are violating one of these constraints.
Make no mistake about it, when that happens, it's on the product
manager. It is your job to ensure that you understand each of the
relevant constraints, and take positive action to work within them.
Marketing
We've already discussed product marketing, which we view more as a
member of the product team than as a stakeholder. But, more generally,
marketing cares about enabling sales, they care about the brand and
reputation of the company, and they care about market competitiveness
and differentiation. Marketing needs the resulting products to be
relevant and compelling, and work with the go-to market channels. So,
anything that you're considering that puts those at risk will be a
major concern.
If what you are proposing to build could impact the sales channel, the
major marketing programs, or is potentially outside of the brand promise
(the range of things your customers expect from your company), then
you'll want to discuss this with marketing and show them prototypes of
what you are proposing before you consider building anything. Work
with them to find ways to address their concerns.
Sales
If your company has a direct sales organization or an advertising sales
organization, then this has a very big impact on the product
organization. Successful products typically need to be designed around
the strengths and limitations of the sales channel.
For example, a direct sales channel is very expensive, and this requires
a high-value product and price point. Or, you may have built up a sales
channel with a certain set of skills, and if your new product requires a
very different set of skills and knowledge, your sales force may
completely reject the product.
If what you are proposing would represent a departure from what the
sales channel has proven
their ability to sell, sit down with the sales leadership and show them
what you are proposing before you build anything, and see if together
you can figure out a way to effectively sell this.
Customer Success
Some tech companies have what's referred to as a high-touch model of
helping their customers, and some have a low-touch model. You need to
understand what your company's customer success strategy is, and you
need to ensure that your products are aligned with that strategy.
Again, if you are proposing something that would represent a change,
you'll want to sit down with leadership and discuss the options.
As a side note, if you have a high-touch service model, these people are
exceptionally helpful for product insights and prototype testing.
Finance
Finance often represents several different constraints and
considerations, not the least of which is whether you can afford to
build, sell, and operate your new product. But, business analytics and
reporting are often in finance, and investor relations and other
concerns may bring their own set of constraints.
If there are cost issues involved, sitting down with someone in finance
and modeling the costs will be critical to demonstrating to leadership
that you have worked out a viable approach.
Legal
For many tech-powered companies, especially those that are working hard
to disrupt markets, legal can play a very significant role. Privacy
concerns, compliance concerns, intellectual property, and competitive
issues are all common constraints related to legal. You can save
yourself a whole lot of time and grief by sitting down early with someone from your legal
team and discussing with them what you are proposing and whether they
anticipate any issues or areas you should be aware of.
Business Development
Most businesses have some number of close business relationships with
partners of various types, usually with a contract behind each that has
a defined set of commitments and constraints. Sometimes these agreements
can cripple your company's ability to compete. Sometimes they are a
huge win. In either case, you need to understand the impact of these
relationships on your products and what you are proposing to do.
Security
We would normally think of security not as a stakeholder, but more as an
integral part of the engineering organization and hence a part of the
product team. However, the issues involving security are so important
for so many technology-powered companies that I think it's useful to
call the area out. If you are proposing anything at all remotely related
to security, you will probably want to bring your tech lead and sit down
with the security leadership early to discuss the ideas and how you will
address their concerns.
CEO/COO/GM
Of course, with every company there is some CEO or general manager that
is responsible for the business unit. They are very likely aware of all
these constraints, and normally they are worried about them. And if the
product manager is not also aware of the issues, or does not have a plan
for dealing with them, the exec is not going to trust the product
manager or product team.
It doesn't take long for a CEO to figure out whether a product manager
has done her homework and understands the different aspects of the
business.
Testing business viability means making sure that the product solution
that your team  is
proposing will work within the constraints of each of these areas. For
those stakeholders that are impacted, it's important that they have a
chance to review the proposal and ensure their concerns have been
addressed.


This is what is really meant by being the CEO of the product.


There's no question that it's hard enough just trying to come up with
a product that your customers love and your engineers can build and
deliver. Many products never get to this point.
However, the truth is this is not enough. The solution must also work
for your business. And I will warn you now that this is often much more
difficult than it sounds.
Many product managers confess to me that this is the least favorite part
of their job. While I understand that, I also explain to them that this
is often what separates the good product managers from the great ones,
and that more than anything else, this is what is really meant by being
the CEO of the product.
Building a business is always hard. You must have a business model
that's viable. The costs to produce, market and sell your product must
be sufficiently less than the revenue your product generates. You must
operate within the laws of the countries you sell in. You must hold up
your end of business agreements and partnerships. Your product must fit
within the brand promise of your company's other offerings.


[CHAPTER 56][Testing Business Viability]


So how does adding hardware to the equation affect everything we've
discussed thus far?
There are some obvious differences, such as different engineering skill
sets, the need for industrial design, and of course, manufacturing still
takes substantially longer than software, although it continues to
improve.
For the most part, however, everything we have discussed thus far still
applies, although there are some additional challenges as well.
Moreover, when hardware is a part of the equation, the discovery
techniques we've discussed are even more important, especially the role
of prototyping.
The reason is because, with hardware, the consequences of a mistake in
terms of time and money are much more severe. With software, we can
usually issue corrections relatively inexpensively. With hardware, no
such luck.
Specifically, there are more technical feasibility risks with hardware,
and there are additional business viability risks. For example, with
hardware we need a much more sophisticated analysis of parts,
manufacturing costs, and forecasting. That said, the necessary
prototyping of the hardware device has been helped dramatically with the
advent of 3D printing technology.
The bottom line is that
hardware products require tackling the value, usability, feasibility,
and viability risks aggressively and raising your bar on the level of
confidence you have before you commit to manufacturing.


With hardware, the consequences of a mistake in terms of time and money
are much more severe.


Discovery for Hardware Products
So many technology-powered products today have a hardware element within
them. From phones to watches to robotics to cars to medical instruments
to thermostats, smart devices are all around us.


If that's the case, first consider whether the idea is potentially
worth investing the necessary time in discovery. If so, then encourage
the engineers to proceed.
One last point on assessing feasibility: I meet many product managers
who hate any product idea that the engineers say they need additional
time to investigate. To these product managers, this means it is already
too risky and time consuming if that happens.
I tell these product managers that I personally love these items for a
few reasons. First, many of our best product ideas are based on
approaches to solving the problem that are only now possible, which
means new technology and time to investigate and learn that technology.
Second, I find that when engineers are given even a day or two to
investigate, they often come back not only with good answers to the
feasibility question but also with better ways to solve the problem.
Third, these sorts of items are often very motivating to the team, as it
gives them an opportunity to learn and to shine.


Many of our best product ideas are based on approaches to solving the
problem that are only now possible, which means new technology and
time to investigate and learn that technology.


I don't want to scare you. With most product ideas that your engineers
review in discovery, they will quickly consider these points and simply
say "No problem." That's because most of our work is not all that new,
and engineers have usually built similar things many times before.
However, there are definitely ideas where this is not the case, and some
or many of these questions can be very difficult for the engineers to
answer.
One very common example right now is that many teams are evaluating
machine-learning technology, considering build/buy decisions, and
assessing whether the technology is suitable for the job at hand---and,
more generally, trying to understand its potential.
Here's some very practical and important advice for you to consider.
Holding a weekly planning
meeting where you throw a bunch of ideas at the engineers---and demand
they give you some sort of estimate either in time, story points, or any
other unit of effort---is almost certain to go badly. If you put an
engineer on the spot, without time to investigate and consider, you are
very likely to get a conservative answer, partly designed to make you go
away.
If, however, the engineers have been following along as the team has
tried out these ideas with customers (using prototypes) and seen what
the issues are and how people feel about these ideas, the engineers have
probably already been considering the issues for some time. If it's
something you think is worthwhile, then you need to give the engineers
some time to investigate and consider it.
The question isn't, "Can you do this?" Rather, you are asking them to
look into it and answer the question, "What's the best way to do this
and how long would it take?"
The engineers will sometimes come back and say they need to create a
feasibility prototype to answer one or more of these questions.


[Do we know how to build this?]
[Do we have the skills on the team to build
this?]
[Do we have enough time to build this?]
[Do we need any architectural changes to build
this?]
[Do we have on hand all the components we need to build
this?]
[Do we understand the dependencies involved in building
this?]
[Will the performance be acceptable?]
[Will it scale to the levels we need?]
[Do we have the infrastructure necessary to test and run
this?]
[Can we afford the cost to provision
this?]


When we talk about validating feasibility, the engineers are really
trying to answer several related questions:


[CHAPTER 55][Testing Feasibility]


For everything new we add, we ensure we have the necessary
instrumentation  in place
to know immediately if it is working as we expect, and if there are
significant unintended consequences. Frankly, without that
instrumentation, I wouldn't bother to roll out the feature. How would
you know if it was working?
For most product managers, the first thing they do in the morning is to
check the analytics to see what happened during the preceding night.
They're usually running some form of test almost all the time, so
they're very interested in what's happened.
There are of course some extreme environments where everything lives
behind very strict firewalls, but even then, the products can generate
periodic usage reports to be reviewed and approved before being
forwarded (via electronic or printed reports, if necessary) back to the
teams.
I'm very big on radically simplifying products by removing features
that don't carry their own weight. But, without knowing what is being
used, and how it's being used, it's a very painful process to do this
when you don't know what's really going on. We don't have the data to
back up our theories or decisions, so management (rightfully) balks.
My personal view is that you should start from the position that you
simply must have this data, and then work backward from there to figure
out the best way to get it.


I still encounter too many product teams that either aren't
instrumenting their product to collect analytics, or they do it at such
a minor level that they don't know if and how their product is being
used.


Flying Blind
Remarkably, I still encounter too many product teams that either aren't
instrumenting their product to collect analytics, or they do it at such
a minor level that they don't know if and how their product is being
used.
My own teams---and every team I can think of that I've ever worked
with---have been doing this for so long now that it's hard to imagine
not having this information. It's hard for me to even remember what it
was like to have no  real
idea how the product was used, or what features were really helping the
customer, versus which ones we thought had to be there just to help
close a sale.
This is easiest to do with cloud-based products and services, and most
of us use web analytics tools, but sometimes we use homegrown tools for
this too.
Good product teams have been doing this for years. And, not just with
cloud-based sites, but also with installed mobile or desktop
applications---on-premise software, hardware, and devices that call home
periodically and send the usage data back to the teams. A few companies
are very conservative and ask permission before sending the data, but
mostly this just happens silently.
We should all be anonymizing and aggregating the data so there's
nothing personally identifiable in there. Occasionally, however, we see
in the news that another company is in trouble for sending raw data in
the rush to market. It seems the press thinks we're tracking these data
for nefarious purposes, but at least with the companies I know and work
with, they're simply trying to make the products better---more valuable
and more usable. This has long been one of our most important tools for
doing so.
The way the overall process works is that we first ask ourselves what we
need to know about how our products are used, then we instrument the
product to collect this information (the particular techniques depend on
the tool you're using and what you want to collect). Finally, we
generate various forms of online reports to view and interpret these
data.


Hopefully, you can see the power of analytics for product teams.
However, as powerful as the role of data is for us, the most important
thing to keep in mind about analytics is that the data will shine a
light on what is happening, but it won't explain why. We need our
qualitative techniques to explain the quantitative results.

Note that we often refer to analytics as key performance indicators
(KPIs).


[User behavior analytics (click paths,
engagement)]
[Business analytics (active users, conversion rate, lifetime value,
retention)]
[Financial analytics (ASP, billings, time to
close)]
[Performance (load time, uptime)]
[Operational costs (storage, hosting)]
[Go-to-market costs (acquisition costs, cost of sales,
programs)]
[Sentiment (NPS, customer satisfaction,
surveys)]


Second, the tools for accessing and learning from this data have
improved significantly. Mostly, however, I see an increased awareness of
the role that data can play in helping you learn and adapt quickly.
There are five main uses of analytics in strong product teams. Let's
take a close look at each of these uses:
Understand User and Customer Behavior
When most people think of analytics, they think of user analytics. That is, however, but one
type of analytic. The idea is to understand how our users and customers
are using our products (remember, there can be many users at a single
customer---at least in the B2B context). We may do this to identify
features that are not being used, or to confirm that features are being
used as we expect, or simply to gain a better understanding of the
difference between what people say and what they actually do.
This type of analytic has been collected and used for this purpose by
good product teams for at least 30 years. A solid decade before the
emergence of the web, desktops and servers were able to call home and
upload behavior analytics, which were then used by the product team to
make improvements. This to me is one of the very few non-negotiables in
product. My view is that, if you're going to put a feature in, you need
to put in at least the basic usage analytics for that feature.
Otherwise, how will you know if it's working as it needs to?
Measure Product Progress
I have long been a strong advocate of using data to drive product teams.
Rather than provide the team an old-style roadmap listing someone's
best guess of what features may or may not work, I strongly prefer to
provide the product team with a set of business objectives---with
measurable goals---and then the team makes the calls as to what are the
best ways to achieve those goals. It's part of the larger trend in
product to focus on outcome and not output.
Prove Whether Product Ideas Work
Today, especially for consumer companies, we can isolate the
contribution of new features, new versions of workflows, or new designs
by running A/B tests and then comparing the results. This lets us prove
which of our  ideas work.
We don't have to do this with everything, but with things that have
high risk or high deployment costs, or that require changes in user
behavior, this can be a tremendously powerful tool. Even where the
volume of traffic is such that collecting statistically significant
results is difficult or time consuming, we can still collect actual data
from our live-data prototypes to make decisions that are much better
informed.
Inform Product Decisions
In my experience, the worst thing about product in the past was its
reliance on opinions. And, usually, the higher up in the organization
the person was who voiced it, the more that opinion counted.
Today, in the spirit of data beats opinions, we have the option of
simply running a test, collecting some data, and then using that data to
inform our decisions. The data is not everything, and we are not slaves
to it, but I find countless examples today in the best product teams of
decisions informed by test results. I hear constantly from teams how
often they are surprised by the data, and how minds are changed by it.
Inspire Product Work
While I am personally hooked on each of the above roles of analytics, I
must admit that my personal favorite is this last point. The data we
aggregate (from all sources) can be a gold mine. It often boils down to
asking the right questions. But by exploring the data, we can find some
very powerful product opportunities. Some of the best product work I see
going on right now was inspired by the data. Yes, we often get great
ideas by observing our customers, and we do often get great ideas by
applying new technology. But studying the data itself can provide
insights that lead to breakthrough product ideas.
Largely, this is because the data often catches us off guard. We have a
set of assumptions about how the product is used---most of which we are
not even conscious
of---and when we see the data, we're surprised that it doesn't track
with those assumptions. It's these surprises that lead to real
progress.
It's also important for tech product managers to have a broad
understanding of the types of analytics that are important to your
product. Many have too narrow of a view. Here is the core set for most
tech products:


Any capable product manager today is expected to be comfortable with
data and understand how to leverage analytics to learn and improve
quickly.


The Role of Analytics
One of the most significant changes in how we do product today is our
use of analytics. Any capable product manager today is expected to be
comfortable with data and understand how to leverage analytics to learn
and improve quickly.
I attribute this change to several factors.
First, as the market for our products has expanded dramatically due to
access globally---and also by way of connected devices---the sheer
volume of data has dramatically increased, which gives us interesting
and statistically significantly results much faster.


In a true startup environment, we don't have much traffic and we also
don't have much time, but we're usually fine with risk (we don't have
much to lose yet).
In a more established company, we often have a lot of traffic, we have
some amount of time (mostly we're worried about management losing
patience), and the company is usually more averse to risk.
A/B Testing
The gold standard for this type of testing is an A/B test. The reason we love A/B tests is
because the user doesn't know which version of the product she is
seeing. This yields data that is very predictive, which is what we
ideally want.
Keep in mind that this is a slightly different type of A/B test than
optimization A/B testing. Optimization testing is where we experiment
with different calls to action, different color treatments on a button,
and so forth. Conceptually they are the same, but in practice there are
some differences. Optimization testing is normally surface-level,
low-risk changes, which we often test in a split test (50:50).
In discovery A/B testing, we usually have the current product showing
to 99 percent of our users, and the live-data prototype showing to only
1 percent of our users or less. We monitor the A/B test more closely.
Invite-Only Testing
If your company is more risk averse, or if you just don't have enough
traffic to be able to show to 1 percent---or even 10 percent---and get
useful results anytime soon, then another effective way to collect
evidence is the invite-only test. This is where you identify a set of
users or customers that you contact and invite to try the new version.
You tell them that it is an experimental version, so they are
effectively opting in if they choose to run it.
The data that this group generates is not as predictive as from a true,
blind, A/B test. We realize that those who opt in are generally more
early adopter types; nevertheless, we are getting a set of actual users
doing their work with our live-data prototype, and we are collecting
really interesting data.
I can't tell you how often we think we have something they'll love,
and then we make it available to a limited group like this and we find
that they are just not feeling it. Unfortunately, with a quantitative
test like this all we know for sure is that they're not using it---we
can't know why. That's when we'll follow up with a qualitative test
to try and quickly learn why they're not as into it as we had hoped.
Customer Discovery Program
A variation of the invite-only test is to use the members of the customer discovery program we
discussed in the section on ideation techniques. These companies have
already opted in to testing new versions, and you already have a close
relationship with them so you can follow up with them easily.
For products for businesses, I typically use this as my primary
technique for collecting actual usage data. We have the customer
discovery program customers getting frequent updates to the live-data
prototype, and we compare their usage data to that of our broader
customers.


While qualitative testing is all about fast learning and big insights,
quantitative techniques are all about collecting evidence.


While qualitative testing is all about fast learning and big insights,
quantitative techniques are all about collecting evidence.
We will sometimes collect enough data that we have statistically
significant results (especially with consumer services with a lot of
daily traffic), and other times we'll set the bar lower and just
collect actual usage data that we consider useful evidence---along
with other factors---to make an informed decision.
This is the main purpose of the live-data prototype we discussed
earlier. As a reminder, a live-data prototype is one of the forms of
prototype created in product discovery intended to expose certain use
cases to a limited group of users to collect some actual usage data.
We have a few key ways to collect this data, and the technique we select
depends on the amount of traffic we have, the amount of time we have,
and our tolerance for risk.


[CHAPTER 54][Quantitative Value Testing Techniques]


The remarkable thing about this kind of qualitative testing is just how
easy and effective it is. The best way to prove this to yourself is to
take your laptop or mobile device with your product or prototype on it
to someone who hasn't seen it yet, and just give it a try.
One important note. As product manager, you need to make sure you are at every single
qualitative value test. Do not delegate this, and certainly don't try
to hire a firm to do this for you. Your contribution to the team comes
from experiencing as many users as possible, first hand, interacting
with and responding  to
your team's ideas. If you worked for me, the continuation of your
monthly salary would depend on this.


As product manager, you need to make sure you are at every single
qualitative value test. Do not delegate this.


When you do this type of qualitative user testing, you don't get your
answer from any one user, but every user you test with is like another
piece of the puzzle. Eventually, you see enough of the puzzle that you
can understand where you've gone wrong.
I know this is a big claim, but I argue that qualitative testing of your
product ideas with real users and customers is probably the single most
important discovery activity for you and your product team. It is so
important and helpful that I push product teams to do at least two or
three qualitative value tests every single week. Here's how to do it:
Interview First
We generally begin the user test with a short user interview where we
try to make sure our user has the problems we think she has, how she
solves these problems today, and what it would take for her to switch
(see Customer Interview Technique).
Usability Test
We have many good techniques for testing value qualitatively, but they all depend on the
user first understanding what your product is and how it works. This is
why a value test is always preceded by a usability test.
During the usability test, we test to see whether the user can figure
out how to operate our product. But, even more important, after a
usability test the user knows what your product is all about and how
it's meant to be used. Only then can we have a useful conversation with
the user about value (or lack thereof).
Preparing a value test therefore includes preparing a usability test. I
described how to prepare for and run a usability test in the last
chapter, so for now let me again emphasize that it's important to
conduct the usability test before the value test, and to do one
immediately after the other.
If you try to do a value test without giving the user or customer the
opportunity to learn how to use the product, then the value test becomes
more like a focus group where people talk hypothetically about your
product, and try to imagine how it might work. To be clear: focus groups
might be helpful for gaining market insights, but they are not helpful
in discovering the product we need to deliver (see Product Discovery
Principle #1).
This testing involves at least you as product manager and your product
designer, but I am constantly amazed at how often the magic happens
when one of your engineers is right there watching the qualitative
testing with you. So, it's worth your pushing to make this happen as
much as possible.
To test usability and value, the user needs to be able to use one of the
prototypes we described earlier. When we're focused on testing value,
we usually utilize high-fidelity user prototypes.
High-fidelity means it feels very realistic, which turns out to be
especially important for value testing. You can also use a live-data
prototype or a hybrid prototype.
Specific Value Tests
The main challenge in testing value when you're sitting face to face with actual users
and customers is that people are generally nice---and not willing to
tell you what they really think. So, all of our tests for value are
designed to make sure the person is not just being nice to you.
Using Money to Demonstrate Value
One technique I like for gauging value is to see if the user would be
willing to pay for it, even if you have no intention of charging them
for it. We're looking for the user to pull out his or her credit card
right then and there and ask to buy the product (but we don't really
want the card information).
If it's an expensive product for businesses---beyond what someone would
put on a credit card---you can ask people if they will sign a
"non-binding letter of intent to buy" which is a good indicator that
people are serious.
Using Reputation to Demonstrate Value
But there are other ways a user can "pay" for a product. You can see if
they would be willing to pay with their reputation. You can ask them how
likely they'd be to recommend the product to their friends or
co-workers or boss (typically on a scale of 0--10). You can ask them to
share on social media. You can ask them to enter the e-mail of their
boss or their friends for a recommendation (even though we don't save
the e-mails, it's very meaningful if people are willing to provide
them).
Using Time to Demonstrate Value
Especially with businesses, you can also ask the person if they'd be
willing to schedule some significant time with you to work on this (even
if we don't need it). This is another way people pay for value.
Using Access to Demonstrate Value
You can also ask people to provide the login credentials for whatever
product they would be switching from (because you tell them there's a migration utility or
something). Again, we don't really want their login and password---we
just want to know if they value our product highly enough that they're
truly willing to switch right then and there.
Iterating the Prototype
Remember, this is not about proving anything. It's about rapid
learning. As soon as you believe you have a problem, or you want to try
a different approach, you try it.
For example, if you show your prototype to two different people and the
response you get is substantially different, your job is to try to
figure out why. Maybe you have two different types of customers, with
different kinds of problems. Maybe you have different types of users,
with different skill sets or domain knowledge. Maybe they are running
different solutions today, and one is happy with their current solution
and one is not.
You might determine that you just aren't able to get people interested
in this problem, or you can't figure out a way to make this usable
enough that your target users can realize this value. In that case, you
may decide to stop right there and put the idea on the shelf. Some
product managers consider this a big failure. I view it as saving the
company the wasted cost of building and shipping a product your
customers don't value (and won't buy), plus the opportunity cost of
what your engineering team could be building instead.


I argue that qualitative testing of your product ideas with real users
and customers is probably the single most important discovery activity
for you and your product team.


Quantitative testing tells us what's happening (or not), but it can't
tell us why, and what to do to correct the situation. That's why we
do qualitative testing. If users and customers are not responding to a
product the way we had hoped, we need to figure out why that's the
case.
As a reminder, qualitative testing is not about proving anything.
That's what quantitative testing is for. Qualitative testing is about
rapid learning and big insights.


[CHAPTER 53][Qualitative Value Testing Techniques]


I believe it's a non-negotiable that we simply must continue to move
our products forward, and deliver increased value to our customers.
That said, we need to do this in a responsible way. This really means
doing two really big things---protect your revenue and brand, and
protect your employees and customers.
Protect Revenue and Brand
The company has built a reputation and has earned revenue, and it is the
job of the product teams to do product discovery in ways that protect
this reputation and this revenue. We've got more techniques than ever
to do this, including many techniques for creating very low-cost and
low-risk prototypes, and for proving things work with minimal investment
and limited exposure. We love live-data prototypes and A/B testing
frameworks.
Many things do not pose a risk to brand or revenue, but for the things
that do, we utilize techniques to mitigate this risk. Most of the time
an A/B test  with 1 percent
or less of the customers exposed is fine for this.
Sometimes, however, we need to be even more conservative. In such cases,
we'll do an invite-only live-data test, or we'll utilize our customer
discovery program customers that are under NDA. There are any number of
other techniques in the same spirit of test and learn in a responsible
way.
Protect Employees and Customers
In addition to protecting revenue and brand, we also need to protect our
employees and our customers. If our customer service, professional
services, or sales staff are blindsided by constant change, it makes it
very hard for them to do their jobs and take good care of customers.
Similarly, customers that feel like your product is a moving target that
they have to constantly relearn won't be happy customers for long.
This is why we use gentle deployment techniques, including assessing
customer impact. Although this may seem counterintuitive, continuous
deployment is a very powerful gentle deployment technique, and when used
properly along with customer impact assessment, it is a powerful tool
for protecting our customers.
Again, most experiments and changes are non-issues, but it is our
responsibility to be proactive with customers and employees and
sensitive to change.
Don't get me wrong. I am not arguing that innovating in enterprise
companies is easy---it's not. But it's not because product discovery
techniques are the obstacles to innovation. They are absolutely critical
to consistently delivering increased value to customers. There are
broader issues in large enterprise companies that typically create
obstacles to innovation.
If you are at a larger, enterprise company, know that you absolutely
must move aggressively to continuously improve your product, well beyond
small  optimizations. But
you also must do this product work in ways that protect brand and
revenue, and protect your employees and your customers.


That said, we need to do this in a responsible way.


I have long argued that the techniques of product discovery and rapid
test and learn absolutely apply to large enterprise companies, and not
just to startups. The best product companies---including Apple, Amazon,
Google, Facebook, and Netflix---are great examples where this kind of
innovation is institutionalized. In these companies, innovation is not
something that just a few people get permission to pursue. It is the
responsibility of all product teams.
But before I go any further, I want to emphasize the most important
point for technology companies: If you stop innovating, you will die.
Maybe not immediately, but if all you do is optimize your existing
solutions, and you stop innovating, it is only a matter of time before
you are someone else's lunch.


The most important point for technology companies: If you stop
innovating, you will die.


Suppose you were contemplating a new feature, perhaps because a large
customer is asking for it or maybe because you saw that a competitor has
the feature or maybe it's your CEO's pet feature. You talk about the
feature with your team, and your engineers point out to you that the
implementation cost is substantial. Not impossible but not easy
either---enough that you don't want to take the time to build this only
to find out later  it
wasn't used.
The demand-testing technique is called a fake door demand test. The
idea is that we put the button or menu item into the user experience
exactly where we believe it should be. But, when the user clicks that
button, rather than taking the user to the new feature, it instead takes
the user to a special page that explains that you are studying the
possibility of adding this new feature, and you are seeking customers to
talk to about this. The page also provides a way for the user to
volunteer (by providing their e-mail or phone number, for example).
What's critical for this to be effective is that the users not have any
visible indication that this is a test until after they click that
button. The benefit is that we can quickly collect some very helpful
data that will allow us to compare the click-through rate on this button
with our expectations or with other features. And then we can follow up
with customers to get a better understanding of what they would expect.
The same basic concept applies to entire products. Rather than a button
on a page, we set up the landing page for the new offering's product
funnel. This is called a landing page demand test. We describe that
new offering exactly as we would if we were really launching the
service. The difference is that if the user clicks the call to action,
rather than signing up for the trial (or whatever the action might be),
the user sees a page that explains that you are studying the possibility
of adding this new offering, and you'd like to talk with them about
that new offering if they're willing.
With both forms of demand testing, we can show the test to every user
(in the case of an early startup) or we can show it to just a very small
percentage of users or within in a specific geography (in the case of a
larger company).
Hopefully, you can see that this is very easy to do, and you can quickly
collect two very useful things: (1) some good evidence on demand and (2)
a list of users who are very ready and willing to talk with you about
this specific new capability.
In practice, the demand  is
usually not the problem. People do sign up for our trial. The problem is
that they try out our product and they don't get excited about it---at
least not excited enough to switch from what they currently use. And
dealing with that is the purpose of the qualitative and quantitative
techniques in the chapters that follow.
Discovery Testing in Risk-Averse Companies
Much has been written about how to do product discovery in startups---by
me and by many others. There are many challenges for startups, but most
important is survival.
One of the real advantages to startups from a product point of view is
that there's no legacy to drag along, no revenue to preserve, and no
reputation to safeguard. This allows us to move very quickly and take
significant risks without much downside.
However, once your product develops to the point that it can sustain a
viable business (congratulations!), you now have something to lose, and
it's not surprising that some of the dynamics of product discovery need
to change. My goal here is to highlight these differences and to
describe how the techniques are modified in larger, enterprise
companies.
Others have also been writing about how to apply these techniques in
enterprises, but on the whole, I have not been particularly impressed
with the advice I've seen. Too often, the suggestion is to carve out a
protected team and provide them some air cover so they can go off and
innovate. First of all, what does this say about the people not on these
special innovation teams? What does this say about the company's
existing products? And, even when something does get some traction,
how well do you think the existing product teams will accept this
learning? These are some of the reasons I'm not an advocate of so-called corporate innovation
labs.


One of the biggest possible wastes of time and effort, and the reason
for countless failed startups, is when a team designs and builds a
product, yet, when they finally release the product, they find that
people won't buy it.


One of the biggest possible wastes of time and effort, and the reason
for countless failed startups, is when a team designs and builds a
product---testing usability, testing reliability, testing performance,
and doing everything they think they're supposed to do---yet, when they
finally release the product, they find that people won't buy it.
Even worse, it's not like they sign up for a trial in significant
numbers, but then for some reason don't decide to buy. We can usually
recover from that. It's that they don't even want to sign up for the
trial. That's a tremendous and often fatal problem.
You might experiment with pricing, positioning, and marketing, but you
eventually conclude that this is just not a problem people are concerned
enough about.
The worst part of this scenario is that, in my experience, it's so
easily avoided.
The problem I just described can happen at the product level, such as an
all-new product from a startup, or at the feature level. The feature
example is depressingly common. Every day, new features get deployed
that don't get used. And, this case is even easier to prevent.


[CHAPTER 52][Demand Testing Techniques]


There are several elements of value, and there are techniques for
testing all of them.
Testing Demand
Sometimes it's unclear if there's demand for what we want to build.
In other words, if we could come up with an amazing solution to this problem, do
customers even care about this problem? Enough to buy a new product and
switch to it? This concept of demand testing applies to entire products,
down to a specific feature on an existing product.
We can't just assume there's demand, although often the demand is well
established because most of the time our products are entering an
existing market with demonstrated and measurable demand. The real
challenge in that situation is whether we can come up with a
demonstrably better solution in terms of value than the alternatives.
Testing Value Qualitatively
The most common type of qualitative value testing is focused on the
response, or reaction. Do customers love this? Will they pay for it?
Will users choose to use this? And most important, if not, why not?
Testing Value Quantitatively
For many products, we need to test efficacy, which refers to how well
this solution solves the underlying problem. In some types of products,
this is very objective and quantitative. For example, in advertising
technology, we can measure the revenue generated and easily compare that
to other advertising technology alternatives. In other types of
products, such as games, it's much less objective.


Just because someone can use our product doesn't mean they will
choose to use our product.


Customers don't have to buy our products, and users don't have to
choose to use a feature. They will only do so if they perceive real
value. Another way to think about this is that just because someone
can use our product doesn't mean they will choose to use our
product. This is especially true when you are trying to get your
customers or users to switch from whatever product or system they were
using before to your new product. And, most of the time, our users and
customers are switching from something---even if that something is a
homegrown solution.
So many companies and product teams think all they need to do is match
the features (referred to as feature parity), and then they don't
understand why their product doesn't sell, even at a lower price.
The customer must perceive your product to be substantially better to
motivate them to buy your product and then wade through the pain and
obstacles of migrating from their old solution.
All of this is a long way of saying that good product teams spend most
of their time on creating value. If the value is there, we can fix
everything else. If it's not, how good our usability, reliability, or
performance is doesn't matter.


[CHAPTER 51][Testing Value]


The point is to gain a deeper understanding of your users and customers
and, of course, to identify the friction points in the prototype so you
can fix them.


Summarizing the Learning
The point is to gain a deeper understanding of your users and customers and, of course,
to identify the friction points in the prototype so you can fix them. It
might be nomenclature, flow, visual design issues, or mental model
issues, but as soon as you think you've identified an issue, just fix
it in the prototype. There's no law that says you have to keep the test
identical for all of your test subjects. That kind of thinking stems
from misunderstanding the role this type of qualitative testing plays.
We're not trying to prove anything here; we're just trying to learn
quickly.
After each test subject, or after each set of tests, someone---usually
either the product manager or the designer---writes up a short summary
e-mail of key learnings and sends it out to the product team. But forget
big reports that take a long time to write, are seldom read, and are
obsolete by the time they're delivered because the prototype has already progressed so far beyond
what was used when the tests were done. They really aren't worth
anyone's time.


[When you first start the actual usability test, make sure to tell your subject that
this is just a prototype, it's a very early product idea, and it's
not real. Explain that she won't be hurting your feelings by giving
her candid feedback, good or bad. You're testing the ideas in the
prototype, you're not testing her. She can't pass or
fail---only the prototype can pass or fail.]
[One more thing before you jump into your tasks: See if they can
tell from the landing page of your prototype what it is that you do,
and especially what might be valuable or appealing to them. Again,
once they jump into tasks, you'll lose that first-time visitor
context, so don't waste the opportunity. You'll find that landing
pages are incredibly important to bridging the gap between
expectations and what the product does.]
[When testing, you'll want to do everything you can to keep your
users in use mode and out of critique mode. What matters is
whether users can easily do the tasks they need to do. It really
doesn't matter if the user thinks something on the page is ugly or
should be moved or changed. Sometimes misguided testers will ask
users questions like "What three things on the page would you
change?" To me, unless that user happens to be a product designer,
I'm not really interested in that. If users knew what they really
wanted, software would be a lot easier to create. So, watch what
they do more than what they say.]
[During the testing, the main skill you have to learn is to keep
quiet. When we see someone struggle, most of us have a natural urge
to help the person out. You need to suppress that urge. It's your
job to turn into a horrible conversationalist. Get comfortable with
silence---it's your friend.]
[There are three important cases you're looking for: (1) the user
got through the task with no problem at all and no help; (2) the
user struggled and moaned a bit, but he eventually got through it;
or (3) he got so frustrated he gave up. Sometimes people will give
up quickly, so you may need to encourage them to keep trying a bit
longer. But, if he gets to the point that you believe he would truly
leave the product and go to a competitor, then that's when you note
that he truly gave up.]
[In general, you'll want to avoid giving any help or leading the witness in any way. If
you see the user scrolling the page up and down and clearly looking
for something, it's okay to ask the user what specifically she's
looking for, as that information is very valuable to you. Some
people ask users to keep a running narration of what they're
thinking, but I find this tends to put people in critique mode, as
it's not a natural behavior.]
[Act like a parrot. This helps in many ways. First, it helps avoid
leading. If they're quiet and you really can't stand it because
you're uncomfortable, tell them what they're doing: "I see that
you're looking at the list on the right." This will prompt them to
tell you what they're trying to do, what they're looking for, or
whatever it may be. If they ask a question, rather than giving a
leading answer, you can play back the question to them. They ask,
"Will clicking on this make a new entry?" and you ask in return,
"You're wondering if clicking on this will make a new entry?"
Usually, they will take it from there because they'll want to
answer your question: "Yeah, I think it will." Parroting also helps
avoid leading value judgments. If you have the urge to say, "Great!"
instead you can say, "You created a new entry." Finally, parroting
key points also helps your note taker because she has more time to
write down important things.]
[Fundamentally, you're trying to get an understanding of how your
target users think about this problem and to identify places in your
prototype where the model the software presents is inconsistent or
incompatible with how the user is thinking about the problem.
That's what it means to be counterintuitive. Fortunately, when you
spot this, it is not usually hard to fix, and it can be a big win
for your product.]
[You will find that you can tell a great deal from body language and
tone. It's painfully obvious when they don't like your ideas, and
it's also clear when they genuinely do. They'll almost always ask
for an e-mail when the product is out if they like what they see.
And, if they really like it, they'll try to get it early from
you.]


Before you jump in, we want to take the opportunity to learn how they
think about this problem today. If you remember the key questions from
the Customer Interview Technique, we want to learn whether the user or
customer really has the problems we think they have, and how they solve
those problems today, and what it would take for them to switch.


We want to learn whether the user or customer really has the problems
we think they have, and how they solve those problems today, and what it
would take for them to switch.


Testing Your Prototype
Now that you've got your prototype ready, lined up your test subjects,
and prepared the tasks and questions, here are a set of tips and
techniques for administering the actual test.


[We usually do usability testing with a high-fidelity user prototype. You can get some
useful usability feedback with a low- or medium-fidelity user
prototype, but for the value testing that typically follows
usability testing, we need the product to be more realistic (more on
why later).]
[Most of the time, when we do a usability and/or value test, it's
with the product manager, the product designer, and one of the
engineers from the team (from those that like to attend these). I
like to rotate among the engineers. As I mentioned earlier, the
magic often happens when an engineer is present, so I try to
encourage that whenever possible. If you have a user researcher
helping with the actual testing, they will typically administer the
test, but absolutely the product manager and designer must be there
for each and every test.]
[You will need to define in advance the set of tasks that you want
to test. Usually, these are fairly obvious. If, for example, you're
building an alarm clock app for a mobile device, your users will
need to do things like set an alarm, find and hit the snooze button,
and so on. There may also be more obscure tasks, but concentrate on
the primary tasks---the ones that users will do most of the
time.]
[Some people still believe that the product manager and the product
designer are too close to the product to do this type of testing
objectively, and they may either get their feelings hurt or only
hear what they want to hear. We get past this obstacle in two ways.
First, we train the product managers and designers on how to conduct
themselves, and second, we make sure the test happens
quickly---before they fall in love with their own ideas. Good
product managers know they will get the product wrong initially and
that nobody gets it right the first time. They know that learning
from these tests is the fastest path to a successful
product.]
[You should have one person administer the usability test and
another person taking notes. It's helpful to have at least one
other person to debrief with afterward to make sure you both saw the
same things and came to the same
conclusions.]
[Formal testing labs will typically have setups with two-way mirrors or closed-circuit
video monitors with cameras that capture both the screen and the
user from the front. This is fine if you have it, but I can't count
how many prototypes I've tested at a tiny table at Starbucks---just
big enough for three or four chairs around the table. In fact, in
many ways, this is preferable to the testing lab because the user
feels a lot less like a lab rat.]
[The other environment that works really well is your customer's
office. It can be time consuming to do, but even 30 minutes in their
office can tell you a lot. They are masters of their domain and
often very talkative. In addition, all the cues are there to remind
them of how they might use the product. You can also learn from
seeing what their office looks like. How big is their monitor? How
fast is their computer and network connectivity? How do they
communicate with their colleagues on their work
tasks?]
[There are tools for doing this type of testing remotely, and I
encourage that, but they are primarily designed for usability
testing and not for the value testing that will usually follow. So,
I view the remote usability testing as a supplement rather than a
replacement.]


Preparing the Test


[If you've established the customer-discovery program I described
earlier, you are probably all set---at least if you're building a
product for businesses. If you're working on a consumer product,
you'll want to supplement that group.]
[You can advertise for test subjects on Craigslist, or you can set
up an SEM campaign using Google AdWords to recruit users (which is
especially good if you are looking for users that are in the
moment of trying to use a product like
yours).]
[If you have a list of e-mail addresses of your users, you can do a
selection from there. Your product marketing manager often can help
you narrow down the list.]
[You can solicit volunteers on your company website---lots of major
companies do this now. Remember that you'll still call and screen
the volunteers to make sure the people you select are in your target
market.]
[You can always go to where your users congregate. Trade shows for
business software, shopping centers for e-commerce, sports bars for
fantasy sports---you get the idea. If your product is addressing a
real need, you usually won't have trouble getting people to give
you an hour. Bring some thank-you gifts.]
[If you're asking users to come to your location, you will likely
need to compensate them for their time. We often will arrange to
meet the test subject at a mutually convenient location, such as a
Starbucks. This practice is so common it's usually referred to as
Starbucks testing.]


Usability testing is typically the most mature and straightforward form
of discovery testing, and it has existed for many years. The tools are
better and teams do much more of this now than they used to, and this is
not rocket science. The main difference today is that we do usability
testing in discovery---using prototypes, before we build the
product---and not at the end, where it's really too late to correct the
issues without significant waste or worse.
If your company is large enough to have its own user research group, by
all means secure as much of their time for your team as you absolutely
can. Even if you can't get much of their time, these people are often
terrific resources, and if you can make a friend in this group, it can
be a huge help to you.
If your organization has funds earmarked for outside services, you may
be able to use one of many user research firms to conduct the testing
for you. But at the price that most firms charge, chances are that you
won't be able to afford nearly as much of this type of testing as your
product will need. If you're like most companies, you have few
resources available, and even less money. But you can't let that stop
you.
So, I'll show you how to do this testing yourself.
No, you won't be as proficient as a trained user researcher---at least
at first---and it'll take you a few sessions to get the hang of it,
but, in most cases, you'll find that you can still identify the serious
issues and friction points with your product, which is what's
important.
There are several excellent books that describe how to conduct informal
usability testing, so I won't try to recreate those here. Instead,
I'll just emphasize the key points.
Recruiting Users to Test
You'll need to round up some test subjects. If you're using a user research group, they'll
likely recruit and schedule the users for you, which is a huge help, but
if you're on your own, you've got several options:


[CHAPTER 50][Testing Usability]


Once we have something that our customers believe is truly valuable, and
we have designed it in a way that we believe our users can figure out
how to use, then we'll typically review the approach with the engineers
to make sure this is doable from their technical feasibility
perspective.
If we're also good on feasibility, then we'll show it to key parts of the business
where there may be concerns (think legal, marketing, sales, CEO, etc.).
We'll often address these business risks last because we don't want to
stir up the organization unless we're confident it's worthwhile. Also,
sometimes the ideas that survive are not so similar to the original
ideas that we started with, and those original ideas may have come from
a business stakeholder. It's much more effective to be able to show
that stakeholder some evidence of what did and didn't work with
customers and why and how you ended up where you are.


In product discovery, we're essentially trying to quickly separate the
good ideas from the bad as we work to try to solve the business problems
assigned to us.


Remember that for many of the things we work on, most or all of these
questions are very straightforward and low risk. Your team is confident.
They have been there and done this many times before, and so we will
proceed to delivery.
The main activity of discovery is when these answers are not so clear.
There is no prescribed order to answering these questions. However, many
teams follow a certain logic.
First, we will usually assess value. This is often the toughest---and
most important---question to answer, and if the value isn't there, not
much else matters. We likely will need to address usability before the
user or customer can even recognize the value. In either case, we
usually assess usability and value with the same users and customers at
the same time.


[Will the user or customer choose to use or buy this?
(Value)]
[Can the user figure out how to use this?
(Usability)]
[Can we build this? (Feasibility)]
[Is this solution viable for our business? (Business
viability)]


Overview
In product discovery, we're essentially trying to quickly separate the
good ideas from the bad as we work to try to solve the business problems
assigned to us. But what does that really mean?
We think of four types of questions we're trying to answer during
discovery:


[Discovery Testing Techniques]


For example, imagine that today you have some sort of live chat--based
help for your customers, but it's only available during the hours when
your customer service staff is in the office. You know that your
customers use your product from all around the world at all hours, so
you would like to develop an automated chat-based system that provides
helpful answers anytime.
You could (and should) talk to your customer service staff about the
types of inquiries they routinely get and how they respond (a concierge
test could help you learn that quickly). Soon you will want to tackle
the challenges of this sort of automation.
One way to learn very quickly and test out several different approaches is to create a
Wizard of Oz prototype that provides a simple, chat-based interface.
However, behind the scenes it is literally you as product manager, or
another member of your team, who is receiving the requests and composing
responses. Soon we begin to experiment with system-generated responses,
perhaps even using a live-data prototype of our algorithm.
These types of hybrids are great examples of the build things that
don't scale philosophy of product discovery. By being a little clever,
we can quickly and easily create tools that let us learn very quickly.
Admittedly, it's mainly qualitative learning, but that's often where
our biggest insights come from anyway.


These types of hybrids are great examples of the build things that
don't scale philosophy of product discovery.


So far, we've explored user prototypes---which are pure
simulations---feasibility prototypes for addressing technical risks, and
live-data prototypes designed to be able to collect evidence, or even
statistically significant proof, as to the effectiveness of a product or
an idea.
While these three categories of prototypes handle most situations well,
a wide variety of hybrid prototypes also combine different aspects of
each of these in different ways.
One of my favorite examples of a hybrid prototype---and an exceptionally
powerful tool for learning quickly in product discovery---is today often
referred to as a Wizard of Oz prototype. A Wizard of Oz prototype
combines the front-end user experience of a high-fidelity user prototype
but with an actual person behind the scenes performing manually what
would ultimately be handled by automation.
A Wizard of Oz prototype is absolutely not scalable, and we would
never send any significant amount of traffic to this. But the benefit
from our perspective is that we can create this very quickly and easily,
and from the user's perspective, it looks and behaves like a real
product.


[CHAPTER 49][Hybrid Prototype Technique]


Today, the technology for creating live-data prototypes is so good that
we can often get what we need in just a couple days to a week. And once
we have it we can iterate very quickly.
Later, we'll discuss the quantitative-validation techniques and you'll
see the different ways we can utilize this live-data prototype. But for
now, know that the key is to be able to send some limited amount of
traffic, and to collect analytics on how this live-data prototype is
being used.
What's important is that actual users will use the live-data prototype for real work, and this
will  generate real data
(analytics) that we can compare to our current product---or to our
expectations---to see if this new approach performs better.


[First, this is code, so engineers must create the live-data prototype, not your
designers.]
[Second, this is not a commercially shippable product, it's not
ready for primetime, and you can't run a business on it. So, if the
live-data tests go well, and you decide to move forward and
productize, you will need to allow your engineers to take the time
required to do the necessary delivery work. It is definitely not
okay for the product manager to tell the engineers that this is
"good enough." That judgment is not up to the product manager. And
the product manager does need to make sure key executives and
stakeholders understand the limitations as
well.]


When creating a live-data prototype, our engineers don't handle all the
use cases. They don't address internationalization and localization
work, they don't tackle performance or scalability, they don't create
the automated tests, and they only include instrumentation for the
specific use cases we're testing.
A live-data prototype is just a small fraction of the productization
effort (in my experience, somewhere between 5 and 10 percent of the
eventual delivery productization work), but you get big value from it.
There are two big limitations you do have to keep in mind, however:


The key is to be able to send some limited amount of traffic, and to
collect analytics on how this live-data prototype is being used.


Sometimes, in order to address a major risk identified in discovery, we
need to be able to collect some actual usage data. But we need to
collect this evidence while in discovery, well before taking the time
and expense of building an actual scalable and shippable product.
Some of my favorite examples of this are when applying game dynamics,
search result relevance, many social features, and product funnel work.
This is the purpose of a live-data prototype.
A live-data prototype is a very limited implementation. It typically
has none of the productization that's normally required, such as the
full set of use cases, automated tests, full analytics instrumentation,
internationalization and localization, performance and scalability, SEO
work, and so forth.
The live-data prototype is substantially smaller than the eventual
product, and the bar is dramatically lower in terms of quality,
performance, and functionality. It needs to run well enough to collect
data for some very specific use cases, and that's about it.


[CHAPTER 48][Live-Data Prototype Technique]


For example, if in my e-commerce user prototype example, I do a search
for a particular type of mountain bike, it always comes back with the
same set of mountain bikes. But if I look closely, they're not the
actual bikes I asked for. And I notice that every time I search, it's
always the same set of bikes no matter what price or style I specify.
If you are trying to test the relevance of the search results, this would not be the
right tool for the job. But if you are trying to come up with a good
overall shopping experience or figure out how people want to search for
mountain bikes, this is probably more than adequate, and it's very
quick and easy to create.
There are many tools for creating user prototypes---for every type of
device, and for every level of fidelity. The tools are mainly developed
for product designers. In fact, your product designer almost certainly
already has one or more favorite user prototyping tools.
It's also the case that some designers prefer to hand-code their
high-fidelity user prototypes, which is fine so long as they are fast,
and they are willing to treat the prototype as disposable.
The big limitation of a user prototype is that it's not good for
proving anything---like whether or not your product will sell.
Where a lot of novice product people go sideways is when they create a
high-fidelity user prototype and they put it in front of 10 or 15 people
who all say how much they love it. They think they've validated their
product, but unfortunately, that's not how it works. People say all
kinds of things and then go do something different.
We have much better techniques for validating value, so it's important
that you understand what a user prototype is not appropriate for.
This is one of the most important techniques for product teams, so it is
well worth developing your team's skills and experience in creating
user prototypes at all levels of fidelity. As you'll see in the coming
chapters, a user prototype is key to several types of validation and is
also one of our most important communication tools.


A user prototype is key to several types of validation and is also one
of our most important communication tools.


A user prototype---one of the most powerful tools in product
discovery---is a simulation. Smoke and mirrors. It's all a façade.
There is nothing behind the curtain. In other words, if you have a user
prototype of an e-commerce site, you can enter your credit card
information as many times as you want---you won't actually be buying
anything.
There is a wide range of user prototypes.
At one end of the spectrum are low-fidelity user prototypes. A
low-fidelity user prototype doesn't look real---it is essentially an
interactive wireframe. Many teams use these as a way to think through
the product among themselves, but there are other uses as well.
Low-fidelity user prototypes, however, represent only one dimension of
your product---the information and the workflow---there's nothing there
about the impact of visual design or the differences caused by the
actual data, to mention just a couple of important examples.
At the other end of the spectrum are high-fidelity user prototypes. A
high-fidelity user prototype is still a simulation; however, now it
looks and feels very real. In fact, with many good high-fidelity user
prototypes, you need to look close to see that it's not real. The data
you see is very realistic, but it's not real either---mostly meaning
it's not live.


[CHAPTER 47][User Prototype Technique]


The main technique used for tackling these types of risks is for one or
more of the engineers to build a feasibility prototype.
An engineer will create the feasibility prototype because it is
typically code (as opposed to most prototypes created by special-purpose
tools intended to be used by product designers). A feasibility prototype
is a long way from a commercially shippable product---the idea is to
write just enough code to mitigate the feasibility risk. This typically
represents just a small percentage of the work for the eventual
shippable product.
Further, most of the time the feasibility prototype is intended to be
throwaway code---it's okay and normal to be quick and dirty with this.
It is intended to be just enough to collect the data, for example, to
show that performance would likely be acceptable or not. There is
usually no user interface, error handling, or any of the typical work
involved in productization.
In my experience, building
a feasibility prototype requires usually just a day or two of time. If
you're exploring a major new technology, such as a new approach
leveraging machine-learning technology, then the feasibility prototype
could very well take significantly longer.
The amount of time the feasibility prototype is estimated to take comes
from the engineers, but whether or not the team takes that time depends
on the product manager's judgment call as to whether it's worth
pursuing this idea. She might say many other approaches to this problem
don't have the technology feasibility risk, so she would rather skip
this idea.
While it's the engineers who do this feasibility prototyping work, it
is considered discovery work and not delivery work. It's done as part
of deciding whether to even pursue this particular approach or idea.
In terms of lessons learned, I have seen many teams proceed to delivery
without adequately considering the feasibility risk. Whenever you hear
stories of product teams that grossly underestimated the amount of work
required to build and deliver something, this is usually the underlying
reason.
It may be that the engineers were simply too inexperienced with their estimates, that
the engineers and product manager had an insufficient understanding of
what was going  to be
needed, or that the product manager did not give the engineers
sufficient time to truly investigate.


The idea is to write just enough code to mitigate the feasibility
risk.


[Algorithm concerns]
[Performance concerns]
[Scalability concerns]
[Fault tolerance concerns]
[Use of a technology the team has not used
before]
[Use of a third-party component or service the team has not used
before]
[Use of a legacy system the team has not used
before]
[Dependency on new or related changes by other
teams]


Most of the time your engineers will review your product ideas and tell
you that they have no real concerns about feasibility. This is because
they have likely built similar things many times before.
However, there are several situations wherein your engineers may
identify a significant feasibility risk involved in solving a particular
problem they are working on. Common examples include:


[CHAPTER 46][Feasibility Prototype Technique]


[The primary purpose of a prototype is to tackle one or more product risks
(value, usability, feasibility, or viability) in discovery; however,
in many cases, the prototype goes on to provide a second benefit,
which is to communicate to the engineers and the broader
organization what needs to be built. This is often referred to as
prototype as spec. In many cases, the prototype is sufficient for
this, but in other cases---especially when the engineers are not
co-located or when the product is especially complex---the prototype
will likely need to be supplemented with additional details
(usually, use cases, business rules, and acceptance
criteria).]


[There are many different possible levels of fidelity for a
prototype. The fidelity primarily refers to how realistic the
prototype looks. There is no such thing as one appropriate level of
fidelity. Sometimes we don't need the prototype to look realistic
at all, and other times it needs to be very realistic. The principle
is that we create the right level of fidelity for its intended
purpose, and we acknowledge that lower fidelity is faster and
cheaper than higher fidelity, so we only do higher fidelity when we
need to.]


The overarching purpose of any form of prototype is to learn
something at a much lower cost in terms of time and effort than
building out a product.


[Similarly, a prototype is also a powerful tool for team
collaboration. Members of the product team and business partners can
all experience the prototype to develop shared
understanding.]


[Realize that one of the key benefits of any form of prototype is to
force you to think through a problem at a substantially deeper level
than if we just talk about it or write something down. This is why
the very act of creating a prototype so often exposes major issues
otherwise left uncovered until much later.]


[The overarching purpose of any form of prototype is to learn
something at a much lower cost in terms of time and effort than
building out a product. All forms of prototype should require at
least an order of magnitude less time and effort as the eventual
product.]


As discussed in Chapter 44, there are many forms of
prototypes. The best choice for you depends on the particular risk being
tackled and the type of product. But all forms of prototypes have
certain characteristics and benefits in common. Here are five key
principles behind their use.


[CHAPTER 45][Principles of Prototypes]


While Fred's quote is as relevant today as when it was first published
(in 1975!), many things have changed, not the least of which is that the
tools and techniques we have for developing prototypes and testing them
have progressed dramatically.
That said, I continue to find teams, and even people I would consider
thought leaders, that have a very narrow interpretation of what is meant
by the term prototype.
When I press people, what I typically find is that they associate the
term prototype with the type that they were first exposed to. If the
first one you saw was used to test for feasibility, that's what you
think of. If the first one you saw was used for usability testing,
that's what you think of.
But there are in fact many very different forms of prototypes, each with
different characteristics and each suited to testing different things.
And, yes, some people get themselves into trouble trying to use the
wrong type of prototype for the job at hand.
In this overview, I highlight the major classes of prototypes, and in
the chapters that follow, I go into depth on each of them.
Feasibility Prototypes
These are written by engineers to address technical feasibility risks
during product discovery---before we decide whether something is
feasible. Sometimes, the engineers are trying out a new technology.
Sometimes it's a new algorithm. Often it is about assessing
performance. The idea is for the developer to write just enough code to
be able to address the feasibility risk.
User Prototypes
User prototypes are simulations. There is a wide spectrum of user
prototypes---from those intentionally designed to look like wireframes
sketched out on paper (referred to as low-fidelity user prototypes) all the way up to those
that look and feel like the real thing (referred to as high-fidelity
user prototypes), where it can be difficult to tell it's just a
simulation.
Live-Data Prototypes
Live-data prototypes are a little more complicated to explain, but they
are a critically important tool for several situations. The main purpose
of a live-data prototype is to collect actual data so we can prove
something, or at least gather some evidence---normally to find out
whether an idea (a feature, a design approach, a workflow) really works.
This typically means two things. First, we need the prototype to access
our live data sources, and second, we need to be able to send live
traffic---in enough quantity to get some useful data---to the prototype.
The key is that we don't want to have to build, test, and deploy a
commercially viable product to do this. That would take far too long,
cost far too much, and very likely yield huge waste. A live-data
prototype costs a small fraction of what it would cost to build a
commercially viable product, which is what makes this such a powerful
tool.
Hybrid Prototypes
There are also many hybrids, which combine aspects of the other types.
For example, when working on search and recommendations in which we're
focusing on relevance, we may need to have the prototype access
live-data sources, but we don't need to be able to send live traffic.
In this case, we're not trying to prove anything, but we can learn a
great deal by observing and discussing the results with the users.
Remember that product discovery is all about coming up with the fastest,
cheapest way to test out our ideas. So, depending on your particular
idea and situation, you'll want to pick the flavor of prototype that
best meets your needs.
While we all might have
our favorites, if you are competing against good product teams, you
are  going to need to be
skilled at each of these.


"Plan to throw one away; you will, anyhow."


Overview
Prototypes of various forms have been around for as long as we've been
applying technology to solve problems. Per the famous Fred Brooks quote,
"Plan to throw one away; you will, anyhow."