THE STARTUP CHAT

[0:00:00]

Steli Efti: Hey everybody, this is Steli Efti.

 

[0:00:03]

Hiten Shah: And this is Hiten Shah. Today on the Start-up Chat, we’re gonna talk about a series of blog posts I’ve been doing with my business partner Marie, around user testing and the value of it. It was a hard one for us to share, and we’ll talk about why. Steli, you wanted to talk about it, so I’d love to get your take, as always, because apparently you’re reading my crap.

 

[0:00:26]

Steli Efti: Yeah, dude. You’re killing it with product habits. Again, for people that are listening. I’m pretty sure that people that listen to the Start-up Chat are already subscribed to Product Habits, but for those that are new that haven’t, go to ProductHabits.com and subscribe. I’ve been on your email list for a long time now, and the content you’re sharing right now is the best content you’ve ever shared, in my opinion. Really it’s-

 

[0:00:51]

Hiten Shah: Wow, thank you.

 

[0:00:52]

Steli Efti: It is killer. It is really, really fucking good. The emails that you send are really good. I know that you and Marie, you’ve kind of gone from a model of considering having a content team that writes these emails, to getting back to the two of you writing these emails, and spending a lot of time back and forth until you get it right. And it really fucking shows. It’s unfortunate, but there’s no shortcut to excellence and to greatness. You have to put in the work. I love the new format right now, the new thing that you guys do where you do these research studies on other companies, right. On their products, on their user onboarding. You do really deep dives and then you share the deep dives and the learnings that you have with brands that are pretty popular. And are teaching even how they could do things better. So I love that content. It’s incredibly insightful. I wanted to pick the last one you did on Grammarly, and share some of the insights you learned there. And then for people that want to learn more, they can just go to ProductHabits.com and read these user research or product research studies in much more depth and detail. But let’s talk a little bit about user onboarding and what you’ve learned through doing these deep dives so far. What are some of the key learnings that you guys took out of it as takeaways?

 

[0:02:16]

Hiten Shah: Yeah, absolutely. A couple things. One, for those of you that don’t know what user testing is, it’s the idea that you can get videos of people walking through an experience, like a product experience, a website, a prototype, an app, and talk about their experience. The key to it, and like Stelli said, they’ll be content on ProductHabits.com. You should subscribe to the newsletter, too, because we share a lot of stuff privately in the emails that we don’t share on the blog. But overall, what user testing is, is this ability for you to see how people react to an experience. That’s the way I like to describe it. It doesn’t have to be something you created. It can be someone else’s. What we did is we user tested Grammarly, Mixmax, and also Duo Lingo. And we’ve done very thorough write-ups about it. What we do is we watch five videos. We ask a bunch of specific questions to the people as they’re going through the experience. And we learn a whole shitload of stuff about how people feel about the experience. Now I’m not gonna get into the whole, of is five people enough, and all that, because just trust me it’s more than enough, especially if you read the write-up. You’ll learn right now about the learning. Grammarly is a really popular product. It’s a Chrome extension. What it does is it helps you with your writing. It’s popular in the sense of it’s had 10 million downloads of the Chrome extension and it has almost seven million active users. Daily active, I believe. They built a brilliant product and a brilliant business. They raised money after being self-funded for a long time, so they have another near and dear place in my heart, and I’m sure Stelli’s. And so, we did these five user tests. Basically it was a user test set of sessions. We did five people walking through the onboarding experience, and we learned a shitload about their onboarding.

 

[0:04:07]

Steli Efti: Let me quickly highlight this. You are paying for this, right?

 

[0:04:11]

Hiten Shah: Yeah, we’re paying for this. Each session costs, I think like $25 to $50. I don’t remember anymore. We’re on the Enterprise account.

 

[0:04:16]

Steli Efti: And let me highlight this again real quick because people are used, and know the concept of buying user research for their own product, their website, their own shit. What nobody does, except you, as far as I know, if go to other people’s product and go, “I’m gonna pay a bunch of stuff to do user research on their product, so I learn from what they’re doing really brilliantly, and what they’re not doing so well, so I can learn from all these things for myself or for other things.” You’re the only one that does this with other people’s products, as far as I’m concerned, or as far as I know.

 

[0:04:52]

Hiten Shah: Yeah. It’s not a popular thing. I think it’s absurd. It’s like, we’re building a Chrome extension coming very soon, and we need to build an onboarding for it. Why would we not just … Why would we, one most people just go copy what other people do as best practice, right?

 

[0:05:08]

Steli Efti: Yep.

 

[0:05:08]

Hiten Shah: Okay, cool. Go copy them. That’s better than making it up yourself, but what’s even better is understanding what works and doesn’t work for other companies that are doing it supposedly well. Think about it. We already know these companies that are doing it supposedly well. Think about it. We already know these companies that I mentioned, have some amount of traction. What we don’t know is what’s working well and not working well for them. And so, the user testing allows you to do that. And these products, they’re not even competitive to what we’re building. They’re just what I would call, what we believe today are best in class experiences for Chrome extension onboarding, or in Duo Lingo’s case, a lot of people rave about their product. So I wanted to understand, “Okay, what does that really mean? How can we learn from it?” I was blown away. Marie and I were blown away by how much we could learn from it. In Grammarly’s case, I’m just gonna go off a little bit. When you go through the experience, one of the things that got me really, really excited about one of the learnings was this. On the home page, if you scroll down a little bit, you get this brilliant demo. It’s this demo where it’s like, here’s a spell checker that you’re used to, and here’s Grammarly and what it can do for you. It’s like night and day. All you have to do is go to Grammarly.com and scroll down a little bit, and you’ll see this brilliant demo. What we learned from the user testing is that people love that demo. And it wasn’t a video. It wasn’t a gif. It was a little more different, and it really caught your eye. You’re like, “Oh. The spell checkers don’t catch any of these mistakes that I’m making, but Grammarly catches these 50 mistakes in this two paragraph piece of writing that I wouldn’t have caught if I use anything else.” People were super highly motivated after seeing that to sign up for the product. Then they start signing up for the product. And they have this thing called personalized Grammarly. It asks you two questions, which is, what do you want to use Grammarly for, or what are you writing for, is actually what it is, and what do you consider your writing level. And after you put that information in, which was very straightforward for people, there’s a next screen that still says personalized Grammarly and it tells you this information. That information, people thought was actually personalized. The truth is, there’s only like nine combinations you can get, because there’s three options for the two things they ask you before that, and you get these nine different options, but everybody only saw one. Honestly, I can be a little more colorful on this podcast, so I will be. It felt like people hated it because they were confused and they didn’t know what it meant. They thought it was analyzing their data because they had already installed the Chrome extension by then, and they were like, “Oh, it’s analyzing my usage.” It’s like, “Actually no. They’re just doing some very basic things and calling it personalization,” but it’s almost like people didn’t care about it. It was almost useless for the experience, but I know from a Grammarly standpoint, it probably helps them highly personalize the things they do for people, the emails they send, and the things like that. I’m almost like, “Why don’t you tell people that? Why don’t you tell people,” and I know I’m going to solution, but why don’t you tell people why you’re asking for the personalization? Honestly, they never tell people why they’re asking for it and what it’s gonna do for them. Instead, they try to tell you, “Oh, this is probably where you’re at because you chose these two options.”

 

[0:08:32]

Steli Efti: Yep. You know, I was going through the Grammarly onboarding as you were speaking, so this is a cool experience for me.

 

[0:08:40]

Hiten Shah: Neat, neat.

 

[0:08:41]

Steli Efti: I was like, I selected the work and whatever the middle level of writer, and then I’m on the personalized Grammarly thing, which shows me now some stats and some stuff with how they can help me. And you’re so right. The first screen I was like, “Oh this is neat. I’m writing for work and yeah, I’m not a pro writer, but not a beginner either. Somewhere in the middle. Cool.” And then the second screen, I was looking and I was like, “All right, this is useless. Whatever.” I didn’t think that this was based on my writing. I didn’t think they were that sophisticated to go and be able to fetch my writing from the past, but I was like, I don’t know, 3.8 mistakes per word typically for a person like … All this is meaningless to me. I was very tempted to just keep … click continue to save your settings and move on with life. So that’s so interesting to hear you describe it as I was going through the different steps myself.

 

[0:09:34]

Hiten Shah: That’s pretty awesome. Yeah, see so you get the sentiment and you’re like, “Crap.” They’re world class at this. They have a ton of active users, but they have a lot of missed opportunity still. And I know, I’ve met some of the people on the team there. These are some of the … I have a high level of respect for the team at this company, and they’re still missing these things because probably they’re not doing this, to be honest with you. Again, not a criticism. It just goes to your point that people don’t do this. I would say that I have a ton more to share, but I’ve shared it on Product Habits. If you have another question, let’s go for it, but people should just be doing user testing. I convinced the UserTesting.com folks to give me a coupon code, so you can get some savings. They give you, I think, three credits for free. And it’s ProductHabits17 is the code. So if you’ve never done UserTesting.com, go to UserTesting.com, use that code. They’re not paying me for this. I just want everyone to build better products, you know.

 

[0:10:32]

Steli Efti: Nice. Nice. I love it. Let’s wrap this episode up with one tip. And then for people that want to know much more, they can go to Product Habits and read the three user testing studies you’ve published. But for people that are like, “All right. I want to do user testing for my product, or for my competitor’s product, but I’m overwhelmed.” Thinking, “How should I go about this?” I mean, you mentioned five interviews is enough. Is there any other tip of what to do or not to do, to be able to get started easily and do this without overthinking it and over-complicating it?

 

[0:11:06]

Hiten Shah: Just do it. I have instructions. Read my instructions and go do it. And even if you don’t read my instructions, just literally go in there, set up a test, and ask people like, “What do you hate the most?”, or, “What confuses you the most?” Make sure you tell them, “Please tell us what confuses you the most about this experience. You will not hurt our feelings. In fact, we want to know what you hate the most. We want to know what confuses you the most because we’re looking to learn from you.” If you just tell them that, in terms of the instructions, that’s all you really need to do. If you want one tip, it’s really about you’re trying to figure out what people really hate. You’re trying to figure out what confuses them. And they tend to not tell you unless you ask them very specifically.

 

[0:11:48]

Steli Efti: I love it. All right. That’s it for this episode, everybody. We’ll hear you very, very soon.

 

[0:11:53]

Hiten Shah: Bye, bye.

[0:11:54]