The focus of the Demand Validation is Product Management episode is about answering the second question of these two standard questions:

• Is this product useable?
• Do people want to use it? – This is Demand Validation

The guest is Steve Cohn, Founder of Validately.

The rest of this blog post contains pretty much raw notes. The notes contain a mixture of direct quotes from the podcast, my own thoughts, and sometimes ideas that came to mind while listening.

Notes

Steve Cohn got a lot of false positives. It begs the question: “How is great innovation brought to market?” It’s relatively easy to define and validate the problem space.

There is a gap between “Is the problem worth it?” and “Solution you have in mind.” Do people want to pay for or talk about your solution?

In comes Demand Validation. Validate the solution space. Figure out ways to run rapid prototypes.

Back to the false positives mentioned above: 50% of the features go unused by customers. “They passed the internal sniff test.” 30% of the work that a product team does is rework. Product teams ask customers “would you use this?” which is a bad indicator of demand.

People are not misleading you on purpose. There’s a gap between the hypothetical desire and committing to the solution. The “cost question” is better. Until you make saying yes cost something, their answer is meaningless. It forces the consumer to make the mental tradeoffs. For the “free products”, instead of costing money we go with time and reputation. So, the three things of possible interest are:

• time – Is the problem deep enough to give their time for free to help you “create” the product?
• money – Money comes in prepaying or signing a contract.
• reputation – Social sharing is the reputation.

If people care about the problem and are compelled by the solution, they will work with you to create your solution.

Money comes in prepaying or signing a contract. Kickstarter is an example for consumers. For businesses, you can pre-sell all the time.

If you get a “no it’s not worth their time / money / reputation”, ask “why.” Knowing why is important. Customers are good at telling you what is not compelling about what is in front of them.

Backlog Prioritization 

Product Management is harder at bigger companies. Some examples include marketing, finance and executive. Marketing is saying there is a feature buzzing out there in the marketplace and we need to do this. Sales say a big customer needs this. Finance says where is the ROI for this feature? Shouldn’t you build another one? Executive says they had a great idea in the shower.

There is a saying mentioned: If we have data, let’s go with data. If we have opinions, go with mine.

Solution: Do Demand Validation Tests at the prototype stage with your backlog. We’re talking about rapid experimentation and getting the data. Then you can respond with the data. Yes, we prototyped it. We tested the idea against the customers and this is what they said. It went poorly so why should we build it? If it tested well, build it ASAP.

A feature prioritization process built on opinions will fail. You need to prioritize based on data.

MVP vs MVE

The true concept of the MVP is brilliant. The problem is when there is a misunderstanding about MVP. He likes to use the term MVE, Minimal Viable Experiment. We’re learning through this experiment. It’s understood that an experiment yields learning. When people hear product, they think growth. So, instead of MVP let’s call it MVE.

We’re in the home-run business. We have to consistently hit home-runs. There’s only one way to do that. The home-runs strike out a lot. So, get lots of trips to the plate. Take lots of swings. Run experiments. You will get a lot of strikeouts. However, you have to do that to get the home-runs. Give the team flexibility to try crazy things.

When Do Product Managers Know They’ve Hit It Big?

Alpha-UX are thought leaders on this. Spit-testing prototypes. They use high volume. They shoot for statistical significance.

The thing you want to understand about validation is that it’s not a guarantee. Lean is about reducing waste. Removing things that definitely won’t work. Prioritize those things that have the higher probability of achieving market success.

Summary

• Do Demand Validation Testing
• Experiment often
• Use data to help prioritize the backlog
• Use the term MVE instead of MVP to set expectations

The focus of the User Research Is Product Management episode is about going beyond simply User Experience. The guest is Jack Cole, a director of design. User Research is critical to product management decision making. It’s beyond just trying to make “cool stuff.”

The rest of this blog post contains pretty much raw notes. The notes contain a mixture of direct quotes from the podcast, my own thoughts, and sometimes ideas that came to mind while listening.

Notes

User research is a through line throughout any type of product lifecycle. — Jack Cole

We focus on users of a certain age or economic background. We do interviews with people, uncover their goals and understand where they are trying to get some additional value. We’re trying to understand the “opportunity space” that is there. We want to create “habit forming” products.

We break it down around the key factors / features that “certain user groups would be be looking for.” We want to go for trends. We uncover them. Next, we go for those identified age ranges, economic backgrounds or areas of the world.

Formal interviews are not the only way. His company has built out friendship groups. It’s more of a casual atmphosphere. They build on a conversation and a level of trust to identify where the opportunities are.

Discover existing pain points and trends. This is done before proposing new ideas and value propositions. Hoping and Praying that things are on target is not fun.

How to Find the Right Users? Who To Talk To?

Envelope yourself in the project. Start gorilla style. Ask around.

Metrics and Data Points

It’s about trying to identify themes in the discussions. Map them out. You can see opportunities based on what is common in the responses. See an opportunity? Build on a hypothesis and on those elements. Objectively identify themes and narratives in data. It’s about quantitive and qualitative information. Shake out the truth vs what is an anomaly. Build out ideas and test.

Common Problem

Trusting in the first interview you hear is a problem. See the themes and through lines. Don’t trust just one voice.

B2Bs count too. Even if the customer base is established, we can achieve better adoption through a deeper understanding of the end user. Come at this through a path of humility. You don’t know everything. There’s always something new that can be pulled out. New innovations can spring forth.

Risks of Not Doing User Research?

• Loss of time, money, and resources.
• If the culture doesn’t embrace it then you get Product Manager burnout.

Evangelizing Tips

It doesn’t have to be time consuming or extensive. Engaging your user community is a form of research.

Reading

• Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration  is the Pixar world. It’s enlightening from a personal and business point of view. (The Amazon reviews are great!)
• How to Kill a Unicorn has interesting points of view.

Podcasts

• Six Pixels of Separation
• Accidental Creative

He also suggests checking out the blog posts at motivatedesign.com

Summary

• Do user research by doing interviews.
• Map out themes from the research.
• Evangelize through community

There’s certainly overlap in this episode with other episodes made by This is Product Management. However, this episode is highly focused which is refreshing. As listed above, the podcasts shared and things he is reading are certainly worth looking into. From what I have seen where I work at CARFAX, understanding the users through user research is a must have. CARFAX gets it. An understood user is a user who will rave about your company.

The focus of the User Experience Is Product Management episode is on User Experience and things that impact such work. The guest is Sarah Doody, a user experience designer.

The rest of this blog post contains pretty much raw notes. The notes contain a mixture of direct quotes from the podcast, my own thoughts, and sometimes ideas that came to mind while listening.

Notes

Sarah works 50% on startups and 50% on companies that are in the market already. She helps them figure out where they should optimize the user experience. She does teaching and writing on the side. She even will be launching some curriculum in 2016 hopefully.

Her Background

She has always been interested in the tech scene. When growing up, she was interested in neuroscience. Specifically, biofeedback. However, that requires medical school. She realized she couldn’t make it through medical school. It’s too gory.

She went to Texas and attended a community college. She did internships. She did web design which was good because her left and right sides of the brain were both used. She took business courses. She took Graphic Design. She did some front end work.

Eventually, she got a full time position in Oregon on a web team doing maintenance of sales sites. She didn’t have a degree yet. At some point, she was asked to do banner ads. It was then she realized this career path was not going to suite her well down the road. He dived into learning about user experience.

She did a startup for five years. After that, she “started her own thing” in 2012.

Intersection of UX and Product Management

User experience involves all the touch points with a product or service. Although there’s the online experience, there could be offline too. There was an example of a frame company. They had a photo of the frame. The product was packaged in beautiful paper. There was a hand written note. It was a great experience. The experience doesn’t end when one leaves the website.

Role Confusion

There’s a difference between UX and UI. UI deals with what does the site look like? It deals with icon colors. UX is stuff before that. How did they get there? What tasks do they want to do? The whole customer journey.

Data Informs How?

Don’t sketch until you’ve talked with people. You don’t have to have a complicated setup with a user lab. You can do research without too much time or budget.

Here’s an example. For one project, she started with doing twitter, message boards, and Quora searches. She tried to answer:

• What are people doing to solve the problem now?
• What websites are they using?
• She spoke with x people in Twitter and x people in Quora.

For her current project, she just got done with a two week research trip to Atlanta, GA. They interviewed people personally in their own environments. They will now take hours of interviews and comb through them and find common themes (Personal thought: Is there an AI opportunity here of finding themes?). They spot themes that come up in each interview. Just spot patterns and understand people’s behaviors at the macro level.

Mistakes Teams Make

Classic mistakes deal with testing on their site. They test too many things at once. Control your variables. Choose one or two things. Otherwise, “the data you get is garbage”.

A tool she likes is Optomizely. It’s simple and you don’t know have to know how to code. She used it on her blog for example. She would test and tweak the location of a social image or button.

Another tool is Crazy Egg. Can see people’s scroll or mouse patterns. It’s good to use a variety of tools. She might use both. That way you get diversified data.

There’s a misconception that you need a massive sample size for interviews. (Like one that needs 100s of people.) She just spoke with 16 people. After the 4th person, they are seeing patterns. The patterns are the same patterns over and over. By the 15th person, you are feeling real done.

If you are doing A / B testing, you need a minimum of a couple of 100 people. It all depends.

Microfeedback

Micro-UX? It’s a small touchpoint that adds a delight factor. Delight in the sense of the experience just works. There’s less thinking required by the user.

An example is the user types “c..a..m.. and predictive text fills in the rest. It helps you get to what you want fast. Another example is with a password form field where there is a show link. You click the show link and you can see the password you just typed in.

Grab the Feedback in a Timely Fashion

It’s important to rapidly grab the feedback from the user after they just did an important task.

For example: She rode in an Uber. A little survey popped up which would let me rate them. That’s micro-feedback. She completed an important task with a product. So, get the feedback close to the point of transaction so you have a chance of getting any feedback at all and it will be better data.

Here’s an example of doing it wrong: Delta. They asked for feedback 3+ days later. They sent an email. They asked “How was the flight?” Click here for a survey. It was 15 to 20 questions long. It’s too long. It looks bad on mobile. They should just send a text message. It should be something like: I saw you landed on Boston. How was your flight: 1 – 5. Done.

Story Telling

She wrote in 2011 Why We Need Storytellers at the heart of product development. Based on a rant on her blog.

She noticed teams that she worked with that the team itself would be over budget, over timeline, bloated with features or sometimes too little features, and the users would not adopt the features. Why is this happening over and over?

It’s like this. There’s a beautiful vision in mind and then what comes out is nothing like what you had in mind in the first place. On Pinterest, you see cakes and decorations and then you try to mimic what you see.

The reason is because we don’t spend enough time in storytelling. It’s because we feel progress writing stuff. However, it’s not real progress.

Spend more time working on the story up front. People will start to embrace it.

Idea: How to apply storytelling? Use storyboarding in the Product Management process. Imagine you are crafting a film. Scene by scene. They are in this particular environment. What are they doing and what’s their problem? Another scene, they have this problem. You have the solution. It helps you think about all the other factors. Example: A location based mapping thing. People are using it outside. It might be sunny. It could impact the colors. So, it might need less words. Consider the whole experience and the whole environment and not just the individual screens.

Collaborate 

She wrote an article called 6 mistakes companies make when working with a UX agency or consultant

Be specific with the feedback. Example of bad feedback: I don’t like orange and green because I don’t like peas and carrots.

Be as specific as possible with the requirements as possible. (Personal thought: What about Agile and incremental / iterative development?)

Be actionable. How to draw out the requirements very very soon is important. Do more of the complex screens up front. It’s going to trigger 1000 questions for them and the teammates.

Don’t just say what you don’t like, say why you don’t like it. I don’t like this because (insert some real business reason).

Don’t hold feedback back. 1/2 way through is too late. Don’t say at the end, I haven’t liked everything up to this point. That doesn’t work. Don’t delay the feedback because that will increase cost.

The cost of change. Changes in UX; factor of 1. Changes in design; factor of 5. Changes in coding; factor in much more.

Don’t race towards the launch date. People see the holidays. A commerce site example: We need to launch this big thing by November. Running toward the launch date causes two things to happen:

1. They end up launching a product with all the features, but all features are 70% quality.
2. You start having to do feature slashing. However, you slashed so many features it’s worthless.

Don’t isolate your developers. Let the developers be involved as much as possible. They are close to the data. They have a good eye for UX sometimes. Look to them for ideas. Have them involved early on and they can help you figure out your time line. Example: Custom chat messaging vs using something out of the box.

Benchmark Question: How does she eat her own dog food?

Amusement park project example will explain it best. She doesn’t have kids. So, she reached out to a few friends. She asked: Ever lost a kid at the grocery store? She used her personal network. She Googled. She looked for products that existed. She spent an afternoon doing the research. She made a storyboard.

In her storyboard, her first screen box was them at a park. Second screen they discover the kid is missing. Next screen they want to take action. What’s the first thing you would want to do? First thing you want to do is report the kid is missing. Keep in mind they need to conserve battery because they are at a park. There was a total of 16 scenes.

That enabled a user flow document. She sketched out what screens would be needed. Similar to sitemap diagrams. Now it’s user flows or screen flows. From there, that allowed her to do wireframes for each screen.

She would bounce between the following as needed: storyboarding, storytelling, user flow / screen flow, wire framing. The main point is that you don’t want to start with wire framing first.

Benchmark Question: How does she get out of the office?

She did formal user analysis. It was too formal. So they developed a process to get information in a less formal and fruitful way.

They created a screener to identify the exact people to talk to. A recruiting firm was hired to find all those people. The quality of recruits is 100% dependent on the screener. Hours were spent on the screener. More time into the screener the better quality of recruits. A discussion guide was created. This consisted of a list of rapid fire questions. The goal was to make interviews conversational and informal yet the questions are in the back pocket.

She talked with people in their own environment. You get more information from people if it is their home or office. For activities on the computer, that helps. They are with the computer they are familiar with. They are familiar with the environment and devices. So she did research in people’s homes and offices.

Benchmark Question: Reading?

She has two books on the go always. For pleasure, My Paris Dream.

Second book: It’s more business oriented. The Shallows by Nicholas Carr. She’s not finished. It’s kind of around the idea that our brains are changing and are not static. What is the influence of the internet on our brains? Multi-tasking is bad. She has learned and confirmed content from there. The internet might not be good for our brains. It’s harder to absorb long form content. Reflecting and focusing on things is a challenge. We’re in distraction mode. We’re bombarded with content. The content may be swishing by our brains. It’s a hard read. However, it ties into her love of studying the brain.

Benchmark Question: Recurring Nightmare

Having projects and clients that lack requirements. There’s a lot of first time founders. People come into product management as UX or from another field. There’s junior product management that is good at the day-to-day. However, there is a product strategy gap. UX is filling that gap. It’s important to identify the requirements and create product roadmaps. As consultant, you want to have those things up front. She may want to figure out how to put the creation of requirements and product roadmaps into her contract.

Summary

The identification of requirements through a process of storytelling is consistent with great advice I have heard from other sources. It sounds like User Experience roles are still being confused with User Interface roles. Finally, the knowledge of how to get valuable insights and feedback from the user are invaluable. Having looked through her website, there are many other gold nuggets to be found at the Sarah Doody website. Also, don’t forget to listen to the User Experience Is Product Management episode made by This is Product Management.

The focus of the User Science is Product Management episode is on User Science. It covers what User Science means, the tools involved and advice around implementing it. The guest is Brent Tworetzky, the Executive Vice President of Product at XO Group. Brent is on Twitter and Medium.

User Science is the study of users. It’s the combination of user research and user analytics. This directly aligns itself with interests that I have. Throughout my career and personal time, I have continuously sought to get closer to the end users of technological solutions. Through the power of enjoyable yet easy to use technology, I like to empower and enrich the lives of others. So, the topic of User Science is an area of deep fascination.

The rest of this blog post contains pretty much raw notes. The notes contain a mixture of direct quotes from the podcast, my own thoughts, and sometimes ideas that came to mind while listening.

Notes

Brent started as an engineer. He didn’t like the day-to-day of programming. So, he got involved in strategy oriented careers such as management consulting. However, he felt “removed from the action.” Product Management lets him put all the things he loves together into one role. As a side note, Brent goes into detail about the Product Management role in his article What Does a Product Manager Do?

Brent has his roots in Product Management from working with others who used to work for eBay, Netflix, and Google. With philosophies that came from those places combined, he has been exposed to different approaches that cover being business case driven, user research oriented, and the use of quick iterations. This all has been foundational to how he applies User Science.

Applying User Science at his organization answers the question of whether to continue to iterate on a product or to go a completely different direction. User Science is a combination of understanding the users’ intent and their user behavior.

Intent vs Action (Behavioral)

The tools involved with Intent vs Action can be expressed in this 2 x 2 grid that Brent provides in his article titled A Product Manager’s Superpower: User Science.

Intent

The purpose of exploring a users intent is to understand what problems the users are trying to solve. Using the intent tools, one is trying to nail down the important problems to solve.

Action / User Behavioral

Once you have the problem to solve, it’s time to switch to the User Behavioral side of things. This is where you can do things like prototype and do usability testing. You can do A/B testing or bucket testing. You can also do aggregate analytics.

From these tools you learn things like:

• Do we have something that still works for what the user wants to do?
• Should we iterate on this or is there stuff to throw away?

Diary Study, A Special User Science Tool

The Diary Study uncovers actual user behavior over an extended period of time. To do the study, they collect a small group of users that agree to share their thoughts and challenges. For example, wedding planning. They check-in every week or every two weeks over a two to twelve month period. The diary captures the needs and challenges of the user each week.

Through this tool, they learn what tools the user uses, what problems they have, and maybe what ideas they might have. Through all of this, the Product Manager can discover whether or not the user has experienced the kind of problem that is being looked for. In other words, if the users are experiencing the same problem again and again, that’s an interesting thing to learn and maybe you’ve found a problem to solve.

Implementation Challenges of User Science

User Science is a field that needs to be learned. It doesn’t come naturally or by accident. When new Product Management team members join their team, they explain and walk through the related philosophies. They ask them to put the tools into their hands and practice right away on practice tests. These new members also get tutorials.

The new team member gets tutorials from the User Research or Product Analytics team. For A/B testing, they use Optimizely. For usability testing, they use UserTesting. They run practice tests and get feedback about those tests. This lets the new team member get familiar with the tools and gain muscle memory about those tools.

On top of all that, their company also brings in guest speakers. The speakers show the capabilities of new tools and how to use the new tools to make “magical” things happen.

Some User Science Tools Need Others to Support

Using some tools requires help from others. For example, diary tests is something that takes a lot of time. His company has a special team to implement those kinds of things.

The trick thing about the tools is that we can’t just tell people about the tools. User Science is a field that needs to be learned. There are four learning curves / stages to User Science.

Four Learning Curves for User Science

One needs to learn:

1. The tools themselves. For example, surveys, one on ones, A/B tests and more.
2. How to do quality tests; Rookie mistakes are common. It’s easy to accidentally bias a test.
3. What test to apply where. Is this the right test for a specific situation?
4. How to use tests in a valuable way. How to gain value from the tests you do. For example, some beginning Product Managers will always use an A/B test for a new idea. That’s their go-to. Brent encourages the Product Managers to use tests to be more “provocative”. Since it’s a test on a small subgroup, the gain is so much greater than the risk.

For Those Getting Started

The User Science Field is advancing quickly. So his advice consists of:

• Hire fantastic people. Help make people great and create an environment to succeed.
• Share a clear vision and mission so all are moving in the same direction.
• Make it clear what success looks like. He recommends Daniel Pink’s book Drive. It covers autonomy, the opportunity for mastery and a rich sense of purpose. From this, one can create a great team environment.
• Focus on outcomes. Make things that matter.
• Communicate, communicate, communicate. There’s no such thing as over communicating especially when companies grow.

Summary

As you can see above, the User Science is Product Management podcast episode is filled with great insight on what makes up User Science, the related tools, and some practical tips on its implementation. I highly recommend you listen to the podcast episode as well as the other episodes at This is Product Management.