Suzanne: I'm so excited to be here, and I have a zillion questions for you. I'm going to try and temper my excitement, and go a little methodically through it if that's all right?
Jason: Sounds good.
Suzanne: Let's talk first and foremost about your background in technology. Can you give us just a flyover of your long career here maybe over the past decade some key experiences you can share with us.
Jason: Sure, I'm happy to. I actually started, I guess, you could say my real career at a company called Silicon Graphics. Probably, younger folks that will be a new name for them, but at the time Silicon Graphics was really one of the big companies in the Valley doing the most innovative things. They did 3D graphic workstations. It was a UNIX based company, so anyone who got hired had to be technical, but what was interesting was I had never studied any coding, or computer science. I remember buying a book called Learn to Publish Through the Web In Seven Days by Laura Lemay. At the time it was a special order book from Barnes & Noble because they didn't have any books on how to do web publishing.
Suzanne: Is this pre-Amazon by the way not to date you?
Jason: Probably about '97, so maybe about the year that Amazon would have been coming out, or thereabouts.
Suzanne: They could have really helped you out.
Jason: Yeah, right, they could have, but in this case it would have been pre-Amazon, and I special ordered the book. Just, basically, a year or two later I found that I was building some websites, I was doing HTML, a little bit of coding on the back to do things like somebody could submit a form, and then it would grab the form information, and send an email to somebody, and you had to do a little bit of coding for that. Lo and behold I ended up having an opportunity to work at Silicon Graphics, and it was because I had that technical background, but thought as a creative person. I had studied liberal arts in college. I think that's what they were really looking for because the Internet was just starting to take shape at that point.
I think that there was an opportunity to have kind of these renaissance minded individuals who were non-developers, non-engineers, non-coders who could see the potential of this new medium called the Internet, and I think there was an excitement about trying to find these people, so I ended up moving up to Silicon Valley. I was living in San Francisco working at Silicon Graphics in Mountain View. This was before the buses, so I got to do the commute. Coincidentally, the Google headquarters today is the old Silicon Graphics headquarters.
I was actually hired as a Web producer, so that was actually my very first title. I was producing an online e-zine for virtual reality developers. The language was called VRML. Essentially, we created an e-zine of content of what people were doing with this VRML technology that SGI was behind. We would have these developers coming, and they could read these articles. We would have weekly features. We would do interviews. We would feature thought leaders not all that different from kind of what we do on GrowthHackers today with growth people. That was kind of where I started. After that I ended up joining a company called Ask Jeeves. They were a search engine. I believe it's now Ask.com.
Suzanne: RIP Ask Jeeves. I remember it well.
Jason: Yeah, exactly right. That was, again, up in the Bay area. That was when I saw the title shift from producer to product manager. I was actually an international PM. Started as a regular PM, became an international PM, and was part of the team that set up Ask Jeeves in the U.K. which turned out to be a really important property for Ask Jeeves.
Suzanne: One of the things that comes up a lot is the interchangeable use of the term Project Manager, and Product Manager. I'm curious you started out in a producer role which is, typically, a project management type role, so when you say you came into product was that somebody sort of said, oh, by the way here's your new title, and then you had to figure out what it meant, or there was a conscious sort of move toward a product world that you understood what that was?
Jason: That's a really great question. I think it was probably the latter. Even though at the time I don't know that there was many product management positions for Internet. I think at that time there was probably product manager roles for like if you were a product manager of an Intel chip, or something like that. There was very much this concept of you have a product you're bringing it to market you collaborate with developers on building that product, marketers to market that product, but at Ask Jeeves it was an Internet company, so it was a little bit amorphous. It was a product management role.
I think that for me I had seen a couple of product managers at Silicon Graphics albeit some were Internet, some were not, so I had an idea of what that meant. I think that for me the difference between let's say that producer/project manager role versus the product manager role was the product manager had to have a little more vision as to what is this product doing? What problem is it solving? How do you create a market, and a demand for this product? What does success look like a year from now? Whereas, I think that in a project management role it's much more tactical than that. I think it's much more how do you keep the project on schedule, on budget making sure that you're going to launch on time, and not run out of money.
Now, I think with that being said any product manager in my opinion needs to have very strong project management skills. I do think the product management role even though it's strategic is also very detail oriented. I also believe that product managers need to be held accountable for the products that they're bringing to market, so if you're going to be held accountable with a responsibility like that it's generally because you're pretty good at keeping things moving, and you can deliver results.
Suzanne: In fact, it's one of the challenges, I think, about the role is this need, sometimes, to oscillate between being in a strategic mindset, being in that sort of visionary mode as you describe, but then having to kind of be down in the trenches, and be tactical in order to get to a launch, or help out wherever there are gaps in the team. Sometimes, we can get stuck a little bit in the tactical because it's comfortable to cross things off of a list. Less comfortable to sort of dream and imagine, and then steer.
Jason: Yeah, I think that that's a really great point, and I call it telescoping. One of the things that I look for when I'm hiring a product manager, or working alongside a product manager is the ability to telescope between different resolutions of detail, so are we in the resolution of detail where we're actually tying to build a product, build a feature, get it shipped, or are we in the resolution of detail where we're really trying to paint a picture of what does this market look like? What is this problem that we're solving? What is the competitive landscape that's out there? Avoiding being myopic. Understanding that you think you do this today, but maybe we're actually doing something bigger. For example, is Uber an on-demand taxi service, or is it really an online dispatch service? It doesn't matter what they're dispatching. That's the business they're in is creating software that can dispatch anything whether it's a taxi, an ice cream truck, or a private plane.
Suzanne: Right, I love that expression telescoping because I've asked many people on the show, and I'll ask you later what they love about product management. A response we hear a lot is no two days are the same. I think it relates to that. Are we kind of zoomed into 3,000 percent, and just focusing on this thing as you describe, or are we sort of way out, so it ties that up nicely. Expect to see it somewhere in a quote later on. I'll be sure to give you credit, though. Okay, Ask Jeeves then what happened?
Jason: Yeah, so then after Ask Jeeves I realized I had kind of cut my teeth in big companies which I'm very grateful for, especially, because those companies were in Silicon Valley. I feel like I can truly, and comfortably say that I learned from the best. I was always in rooms where everyone was smarter than me. I think that that's something I still try to do, and I think everyone should try to join teams where you're not the smartest person on the team. You want to know that everyone is super smart, and everyone has that self-realization that they're amongst incredible peers.
The Internet was coming of age, right? This was kind of around the year 2000. We had had the bubble burst, but after the bubble burst I realized I didn't want to go back into working for big companies. I decided I really wanted to try something that was more of a startup, so it was more that idea of maybe going from zero to one as opposed to previous roles where you're coming into a company where they already have a roadmap, they already have brand recognition. You may be scaling a product, or you're bringing a product to market where there's already a branding distribution behind it which is a different challenge, right?
Suzanne: For sure.
Jason: In this case I wanted to go the startup route. I've been doing the startup route for many years now, albeit different levels of startup. That could be seed level startup to Series A funded startup to Series B, and beyond funded startup. When I joined Sean Ellis about four years ago my co-founder Sean I had actually started as an adviser. Basically, he was working on something called CatchFree. I had started as an adviser, and I was actually in Europe at the time. I was working my way through some of the Index Ventures portfolio companies over there bringing product leadership principles to Europe. It was something that was kind of new to them.
Sean said, "Look, if you ever find yourself back in California it would be great to work together." I was ready to come back to California, moved back here, and about a week after I moved back here we ended up buying what was KISSinsights at the time from the KISSmetrics team. We rebranded KISSinsights Qualaroo, and then, basically, scaled Qualaroo from something that was small to by order of magnitude ended up increasing the annual revenue for that.
Suzanne: Qualaroo was just recently sold.
Jason: Yeah, so Qualaroo was sold earlier this year.
Suzanne: You get to put “successful exit,” now, next to your name.
Jason: Right, exactly right.
Suzanne: That's the new esquire.
Jason: Yes. We did sell Qualaroo. We even got our little placard that goes on your desk which is something that happens when exits take place in Silicon Valley which is kind of fun. The idea behind Qualaroo, Sean is, obviously, one of the big thought leaders behind growth marketing, growth hacking. He coined the term growth hacking, and a big part of his process, but I would, obviously, say any process, but certainly Sean has done an amazing job of articulating the importance of it is you have to listen to your customers. Your have to listen to your users, and a big part of his process was surveying, right? He would like to run these surveys.
He had a couple of surveys that he follows that if you find Sean's presentations online he goes into detail about some of the questions that you can ask, and some of the answers that you're looking for, but this idea of listening to your customers in their own words allow them to articulate things like is this a must-have product, and if so why, or why not? How disappointed would you be if this product went away tomorrow? Then using those findings to be able to message, and create demand for the product through the lens of the people who are most passionate about your product.
KISSinsights, like I had mentioned which was borne out of KISSmetrics was really unique in the sense that it allowed marketers to run what I would call a micro-survey because we're talking one or two questions, but at users who are on your website using your application. I believe they now support mobile, so you could also do this through mobile apps, but most importantly you don't know the identity of those users, so unlike a SurveyMonkey survey where, generally, you're sending that survey via email where you already have to have a relationship with that user whether they're just looking, or whether they're a paying customer. If you don't have that email address you can't really deliver the survey.
What was really great about Qualaroo was that you could get these insights from people at different points within the product, so maybe people who had not yet become customers versus people that just became customers versus people that are already habitual customers. By targeting these micro-surveys you could start to get these really rich actionable insights around those very distinct segments.
Suzanne: It's funny that you bring that up. Just last night I was teaching a class on metrics, and talking about kind of pirate metrics in particular, and this idea of if something is broken at a phase within the funnel all work stops, and you got to sort of get your archeologist brush, and start digging around and looking for it. It sounds a little bit like what you're saying is that Qualaroo is the tool that could kind of help you find that problem.
Jason: Exactly. In fact, I would say that there's kind of the qualitative toolset, and the quantitative toolset. So the quantitative toolset would tell you what's happening. Like you had said the funnel is broken, right? You can see that through your quantitative analytics. You can start to see what does my funnel look like? Where are the leaks? Where are the leaks in my bucket? Where are the things that are clearly broken from just a pure quantitative point of view. Then the qualitative starts to shed light on the why it's happening, right?
For example, let's say you have an eCommerce site. I think that one of the use cases that I always thought was really interesting for Qualaroo was if you have a user who just converted for the first time they just made their first purchase popping up a one question Qualaroo survey what was the one thing that almost prevented you from purchasing today? Because, oftentimes, when they're making that first purchase there's a little bit of anxiety because they've never purchased from me before. Maybe they're not sure if the shipping is really free, or if they're going to have a chance to review their order, so by hearing from users’ own voices of here's the one thing that almost made me not purchase you can really start to get some ideas for things that you should be testing. Then, of course, you use your quantitative analytics after you've made some changes to see whether or not that made a difference.
Suzanne: Right. Fascinating. I want to go back to something that you said earlier when you were talking about your history in this journey, and that is how the product manager role changes depending on the type of organization that you're in, so you've worked in larger scale organizations as you described that are sort of a little bit more, well, they're businesses that have been built. There's proper departments that have been stood up, so probably they're subject to some of that fragmentation, or siloing effect that can happen. The roadmap has long been put in place by somebody who preceeded you, and/or more senior to you versus that startup piece which is way more about tinkering and tinkering until you find what works. What can you tell us because I think we're fascinated with startup culture. That's like the sexy thing. It's less sexy to talk about just like safely steering a company through the maturity phase, or the post product market fit era. What does product management look like in a more established company?
Jason: Sure. I think that before I even maybe talk into what could be some of the differences which I think is maybe what you're asking I'd like to start with what are the commonalities, right? What are the commonalities, in my opinion, of a product manager regardless of size of company, or how mature the product is. I think that what is common is being obsessed with the customer. I think that you absolutely have to be obsessed with listening to your users, listening to your customers. Like for myself I try to talk to a customer at least every day. I don't think that that should change even if you're part of a more mature product, or more mature team. Granted in larger companies it may be a little harder to understand how to orchestrate that. You might have to work through other departments, but I think that the importance of keeping the pulse on how the user is getting value from the product that you've delivered.
Value could be different for let's say a consumer product versus a B2B product, but also, keeping your finger on the pulse of understanding what are the things that they love, what are the things that drive them crazy? A lot of times they have ideas themselves of how the product can be better, and they love to talk about it. I think that a lot of people love the idea of talking to somebody who is in a position to make a change on the product, and they just want to say, gosh, if you guys just had this one feature it would be really amazing.
Now that being said the product manager needs to have very good filters. Understand that you may be looking at a sample size of one, but you do start to see patterns after a while. Once you've talked to enough customers you're going to start to hear that the same things come up. Usually, I find that they map to things that you as a PM have already been feeling yourself about the product, so it becomes really great validation that maybe it's time to figure out a way to get this on the roadmap, or an MVP of it on the roadmap.
As far as how things are different, I think, this starts to touch upon maybe some of the challenges of product management. I think as the organization gets bigger you have more opinions, right? I think one of the nice things about being in a startup is that you have a very small number of opinions. If you're in the product role you have a lot of influence over the things that are happening. Certainly, if you're a founder or co-founder you have even more influence, but then I think that as the organization scales you're going to start to bring in a lot more smart people.
The thing about the product that's unique compared to let's say other roles in the business is oftentimes everybody is a user, certainly, in a consumer product, right? I would have to imagine that everybody working at Facebook is a Facebook user, so in a sense everybody is going to have pretty strong opinions about what they love and hate about the product, but it's the product management team themselves that are really in that position to find the roadmap, determine what are the things that are important. It's a pretty big responsibility to be able to stand behind your own roadmap, and being able to justify, and rationalize.
Now the interesting thing about a startup is because you're doing a lot of experimentation you're trying to get things right even though it's a smaller team a lot of times you're going to have these ideas that are potentially disruptive because if you're not seeing much of a change through some of the incremental changes that you're making to the product you may have to do something that's a little bit bolder. It's important that those bold ideas get socialized, I think, for a couple of reasons. One of them is you should sanity check the logic behind your idea. Can you rationalize this, or is it really just coming from the gut? If somebody is going to tell you you're crazy, or that's a really bad idea let that happen in private.
Second of all, assuming that there is something to that idea then you're going to be able to feel a lot more comfortable not only talking about it in a group setting when you know that half, or all the people at the table have already heard the seed of the idea in some form or another, but you're also going to have a more productive meeting because instead of being in sort of a sales mode you can kind of jump right into we've already talked about this idea in one form or another, and let's see if we can put our heads together as to what the MVP looks like to validate this.
Suzanne: You're painting a picture of product management in these organizations a little bit like survivor where you're just sort of going off, and creating all of these micro-alliances, and then when we come back together at sort of the final vote of it, so we've all discussed this.
Jason: What's funny about it is that I do worry that I don't want it to sound political. Now one of the reasons I was always attracted to startups is because they have a tendency to be less political environments than larger companies. Of course, it depends on the culture of the large company. I don't like to think of it as politics. I like to think of it as just good principles of human interaction. I think part of the creative process, certainly, in collaborative creation is there needs to be that socialization of ideas. I think, otherwise, it's too easy to bruise people's egos, or people feel scared to speak up because they don't want to disagree with somebody who is maybe of a different title than themselves, or what-not. Hopefully, it doesn't sound political because it's not meant to. For me it's much more about that collaborative creative process.
Suzanne: It is, I mean, all kidding aside it's stakeholder management, right?
Suzanne: The truth is, and we know this about product management you're negotiating so many different types of thinkers, and so many different types of personalities that when it comes to getting people to row together there is a little bit of art in creating alignment, especially, because a lot of the times you don't have the power you just have the responsibility.
Jason: That's a great point. One of the things that I learned very early on is that the product manager for me manages through influence. Oftentimes, it's a matrix type of role where you may or may not have any direct reports. You're relying on other people to help you bring this product to market, but those people, oftentimes, don't report to you, right? For example, if you're working with a designer to bring a product to market that designer might report to a head of design. If you're working with an engineering team to bring a product to market they report to the head of engineering, so I do think that it's really important to exhibit, again, I think they're just good leadership skills. I think the product manager needs to be a leader. I believe it's the most entrepreneurial role in a company regardless of size, and oftentimes product management is a great path to see your founder because in a sense there's a general management skillset that needs to be there if you're going to be an effective product manager.
Suzanne: I love that you brought up before the reality of being users of products kind of inside. One of the things I say a lot to my students first class is you're already ready to think like a product manager because you're a user of products which you've sort of echoed. Another thing that I say much later in the course is don't fall in love with your own solution which just, again, speaks to the importance of kind of customer discovery, customer development ongoing using tools like Qualaroo, but this idea that if you happen to be a user of products, and your product that that's a double-edged sword because being internal you're already predisposed to thinking you know what's right. That's why we do the customer check. It's like am I thinking for myself, or am I thinking for them?
Jason: Get out of the building.
Suzanne: Get out of the building, exactly, but there's an extra layer when as you say you're a Facebook user because you feel really connected to the customer experience. I mean, I'm not sure that it's a question.
Jason: I was going to say I hope it's not a question because I don't have the answer. Like you said product management is an art and a science. I've mentioned the word renaissance, I think, earlier in our talk. It is kind of that renaissance role. You have to be enough of a scientist that you use instrumentation, and analytics to understand what's happening. You're talking with engineers who are scientists by skill, but then there's an art form to it, and the art form is the ability to lead and manage to influence the ability to empathize with users, but also do a good job of not being too biased.
Suzanne: Right. Yeah, I think the reflection is just simply that just when you thought it was challenging there's another layer of what makes it challenging.
Suzanne: All right, we are here at GrowthHackers, and it's not really a new venture, but I guess I want to say it's new because you're no longer with Qualaroo, you've sold that. For the benefit of our listeners how do you define growth hacking?
Jason: Growth hacking for me it's really taking an objective based approach to experiment driven changes, if you will. I think, first and foremost, a product really needs to have product market fit, and hopefully, pretty strong distribution before you're really ready to start going into the growth phase. If you don't have product market fit all you're doing is growing a leaky bucket, so what does product market fit mean? That's almost a separate podcast in itself, but I think that there are a couple of indicators of when you may have product market fit. Some folks will tell you you'll know it when you see it, or when you have it.
I think really one of the big indicators is retention. You're looking at retention cohorts for the people that come into your product, and if you have really strong retention cohorts, and you find that you have this core group of users that not only never go away over time, but they continue to engage with the product over time that's a good indicator. If you've already started to sell your product, if you're finding that your paying customer base is growing, and you're retaining those paying customers that's also a good indicator that you've reached some level of product market fit.
Once you have that now you're ready to do growth, so growth hacking for me is taking that product that already has fit at least with a part of the market, and looking at all of the leverage points that you have within your funnel. Everything from how you're going to acquire new users to how you're going to activate new users, how you're going to monetize those users to get revenue, how you're going to retain those users. Also, how you're going to have those users act as a referral channel for you.
Actually, what I've just laid out is something that we're big believers in here at GrowthHackers which is Dave McClure's pirate metrics for startups. He calls it that because it's Aarrr like a pirate, right? It's the AARRR the Acquisition, Activation, Revenue, Retention, and Referral. For me growth hacking is creating that growth model to understand where your leverage points are, deciding which part of the funnel to focus on that's going to give you the most leverage, so in another sense kind of what you had mentioned like where is maybe a part of your funnel broken, and maybe it makes sense to focus on that first.
The beautiful thing about the Internet, or any networked products is that you have very deep analytics, so you don't have to speculate where things are broken. You can actually start to see where things are broken. Then taking an objective based approach to how you're going to grow that through experiments. An objective is you have OKR's which is out there. We use the term objective here. We use objective, actually, in the GrowthHackers product. Part of our product allows you to manage this experimentation process. The attributes of an objective is what is it that you want to achieve, right? What is your goal? What is the key result that you're going after meaning like from a numeric point of view like if you're trying to move a conversion rate let's use numbers. Let's not just say I want to increase it.
Suzanne: How does everybody know when we achieve the goals we like to say.
Jason: Exactly, right, how are we going to know, and then what is the timeframe, right? It's not ambiguous. For example, we want to double our conversion rate in our shopping cart funnel by the end of the quarter. This would be an example of an objective. Once you have that objective, in my opinion, the entire team can rally around that objective. I think that it's actually something that's cross functional, so I think you could actually say that growth hacking really happens across the company because sales can play a role in achieving that objective. Customer success can play a role in achieving that objective, and then certainly products can play a role because a lot of times in today's products one of your best opportunities to influence user behavior is within the product itself.
You have this objective. You now start to have let's say some ideas around how you can achieve that objective. Then, finally you need to prioritize those ideas. You're going to prioritize those ideas based on having a hypotheses. What you expect the difference to be and why. If you're using your numbers correctly you're going to have some idea as to why it should actually move the needle. Then, finally, once you've designed that experiment you're going to execute it, release it, monitor it, measure it, and then learn from it. Then the process starts, again, from the beginning based on what you've learned. If things go well you're going to find that you're growing, but it's really taking this very methodical approach. It's really the scientific method, and then there's even elements of, I think, product development ande scrum in there. We actually use the term growth sprint. Our growth sprints are weekly. Our goal is to release three experiments every week.
Suzanne: I'm going to come back to that. Tell us what is GrowthHackers.com then? For those of us who don't know.
Jason: Sure. GrowthHackers.com there's a couple of parts to it. GrowthHackers.com is a place for anybody whose interested in growth marketing to come, and it could be everything from discovering some ideas around let's say different channels, or reading case studies of how some of the larger companies have grown. Being able to do AMA's with thought leaders who oftentimes are working for very fast hyper-growth companies. Posting your own questions what you can do by name, or anonymously, connecting with other people of this community, if you will, this community of growth marketers. Then last year we introduced what we call GrowthHackers Projects. This is our platform that allows teams to collaborate around that growth process that I just talked about.
Suzanne: Okay, so it's essentially a software tool not maybe unlike that you mentioned scrum, not unlike a Pivotal Tracker, or JIRA, but for the express purpose of bringing transparency, and collaboration to the growth hacking initiative within an organization.
Jason: Exactly right, and then I think that part of our vision then becomes how do we intersect that with the network, and leverage the network, the network consisting of this community of people. A place where ideas are exchanged. People with different skillsets and backgrounds. I think that there's a lot that we can do with those two components.
Suzanne: On this topic of growth I want to tie it back to product management. One of the things that we know about product management is that it can be challenging, and exciting for the same reason which is integrated knowledge requirement. I think you spoke about this earlier, so you need to know business, I put in parentheses. You need to understand design and usability. If you're in tech you need to know how software gets built. Sometimes, an organization seeking a product manager to hire may look for a PM who skews heavily toward one domain, so you see technical product manager. You see marketing product manager. You have a talk that you give about an emerging type of product manager called the growth product manager. I think everyone listening is going "There's another type of product manager I have to learn to become." Tell us what is that position, why is it important, and how is it different from just plain old product manager?
Jason: Sure. For me the PM of growth is somebody who has the ability to create that growth model that we talked about, understanding where are the leverage points, what does the funnel look like at this point? Helping the team define these growth objectives, and then working with the team to design these experiments.
I want to say working with the team that's assuming that you are bigger, but remember part of it is that this growth process really only kicks in after you have product market fit, right? Generally speaking I would imagine a PM of growth is at a company that already has some traction, possibly even post Series B if we're talking startups. There's somebody who is rather than being focused on shipping product, and shipping value the PM of growth is focused on shipping growth, and results which is inherent in the fact that the value is already there. What they're doing is getting users to do more of something.
A lot of times that's going to be influencing user behavior through these experiments. One of the reasons I advocate this separate role for PM of growth is because, oftentimes, the highest leverage points happen within the product, therefore, I do believe that there's going to be engineers that are going to be needed to do these experiments. There's going to need to be analysts that are involved in doing these experiments, and you're going to need to have people that are let's say good at copyrighting, good at design, and maybe some other resources that need to be pulled in depending on the nature of the experiment, but just as you would have a roadmap for a product management role where you're shipping value and extending the value proposition of the product this PM of growth has this roadmap of growth experiments.
Of course, that roadmap is changing as you learn and you discover, but what I found is that if you have the same product manager, or the same product team trying to both deliver, extend the value of the core product, and grow the metrics of how users are already using the product that's a lot to juggle. I think that it can be done, but I think that it can be more effectively done if there's that sense of ownership.
Suzanne: Right, the way you're describing it it's like a separation of concerns to say, look, we have something that is in market, and is stable. We have to be delivering kind of day-over-day, week-over-week the value that we've already transacted with our existing customers, and in the background this is like going back to your sort of telescoping. Telescoping?
Suzanne: My mind went to periscoping, and I thought, no, we didn't plug periscope here. This telescoping idea of even though things are going well right now, and there's a day-to-day requirement to support that we need to be thinking about what's next, whether that's maximizing retention as you talked about, whether that's finding a way to create some virality through referrals, et cetera, so it's interesting that there's a separation, but the product manager of growth is in their own sort of separate experimental wing of the building. Like, don't worry about us. We're just going to be hacking stuff.
Jason: Exactly, I mean very much so is that it is kind of that team that they're self-contained. That's the word that I'm looking for. I mean, oftentimes, I think the strongest growth teams will have that sort of product manager, or business person who is, again, leading the vision process, helping design the experiments. Then you have the engineer who’s able to build what's needed to run that experiment. You have a designer who can do any of the design requirement, and then that PM may function as their own analyst, or depending on how complex the model is, or how much data there is you might even have an analyst on that team.
As an example, if you're trying to increase ridership for an app, I would envision that being a PM of growth because in this case you're taking something that users already do today meaning that the product already exists and fulfills a function, and delivers value. Now you're trying to figure out how can you get them to do it more, right? That's going to be a different set of experiments, and designs at the product level then how do I make the product so that now you can get rides with different types of vehicles, or with different types of drivers? I would consider that to be kind of more core product, if you will. I definitely see a distinction between those two, and it becomes more and more distinct as the business grows.
Suzanne: I think this integrated knowledge requirement it can definitely be something that's scary about staring down the barrel of coming into product management, or being in it, but it can also be a really exciting invitation. Another thing that I talk a lot about with the guests is that every PM sort of has their unique blend of skillsets, so if you love product, and you love customers, and you love the idea of creating value for people, but you're just like really obsessed with like how can I break this thing wide open, and it sounds like growth could be the specific sort of path of product management that's for you.
Jason: Yeah, I think that you're absolutely right. I think that people do have a tendency to gravitate towards certain types of product management. Like you said there's senior product managers. I mean, for example, I could never be a product manager for let's say semiconductors because a product manager would have a very engineering heavy background. You have these product managers of growth for me that would be somebody who is probably more akin to a marketing type role. Somebody who may have been a marketer 20, 30 years ago because a lot of times you're using the power of persuasion to try to get the user to do something, and they could be persuaded through different calls to action. They could be persuaded through different language.
Things as subtle as share versus invite like very subtle languages. One of the people that I'm a believer in is Dan Ariely, and Predictably Irrational. I mean he's written an entire book, and teaches entire courses about how to influence users behavior, and customer behavior, and people behavior just by the way that you're presenting information in the words that you're choosing to use. Somebody who also does something similar is Nir Eyal has his book Hooked, and his hooked model which is how do you design, literally, engineer a model where you're hooking users? You're effectively getting them to habitually use something.
Suzanne: Yeah, I love that book. It does really read like a diabolical playbook.
Jason: It does. Use with caution, right.
Suzanne: Then he speaks to that with his manipulation matrix at the end, but you're reading, and then you're going "Is this allowed to be out on the shelves because..."
Jason: I think that these things are great because the reality is that these are systems. These are processes. These are things that are very deliberate. There's an art to making it work, but I think that product manager of growth person, again, has probably a lot of elements from a marketing background because they really don't have to understand what's technically possible so much as they have to just understand how can I design some experiments to see if I can get users to behave in a different way versus let's say a core product manager that probably veers a little bit more towards the technical in an Internet, or app company because you're shipping products, you're making decisions around MVP, and you're making decisions around is there an easier way.
I think there's elements of that in this PM of growth role like what is that minimum viable experiment, if you will, right? We don't want to over invest in building an experiment if we don't know it works yet, but I think that more core product PM is probably a little bit closer to the technology side of the spectrum. What I think they both have to have in common is very comfortable with analytics.
Suzanne: Right. This is a loaded question I realize, but when you're doing rapid experimentation you shared with us that you do hear growth hackers sort of three experiments per week. What is the right amount of time to look at the data before you're ready to make a pivot, or derive a clear insight from that because I think that's a question that comes up for a lot of people is that is it a day? Is it a week? Is it a month?
Jason: I think that the answer is it depends.
Suzanne: That's the official product management answer for everything.
Jason: It is. I think that there's a couple of things going on. Obviously, the experimentation process is a data driven process. It's the scientific method. Something that you'll read and hear a lot about is you look at A/B testing, and multivariate testing is the importance of reaching statistical significance of your test before you make a call. I completely agree as somebody who had the privilege of working for a company where we had enough traffic that we could truly run every experiment, and then make a call within 24 hours as to whether or not we had statistically, significant answers to whether it worked or not.
That was great, and if you have the traffic that's how you should be making the call, but if you don't have that kind of traffic then, oftentimes, you're going to be launching experiments, and you're going to have to look at the user behavior, and kind of look for directional data. Now, of course, the danger of directional data is because you didn't have a statistically, significant sample you may make a decision prematurely, and then only find out later on that you made the wrong one, but from my personal experience, oftentimes, you can make some directional decision.
If you're talking to your customers because you're a smaller set you can even use some qualitative input from some of your most passionate users to start to understand is this an improvement or not? Then, of course, you can look at before and after. What did life look like before we rolled out this change? What did life look like after this change? Even though that's not a strict A/B test that's more of a time series test. I think that you can still get some directional data that, yeah, we're seeing that more people are converting, or more people are getting through the onboarding flow, or we can see that more people are inviting people, or the quality of things they do on our product is higher with this flow than it was with the previous flow, and be able to tell a story with examples.
Suzanne: This kind of wavering between the importance like the real importance, and necessity of numbers as you say sort of quantitative analysis in this experimental method which is so much the root, the core methodology, I think, of product management. Then as you've said multiple times now not abandoning qualitative entirely, right? Remembering that the numbers can tell you a lot, but I think just like we talked about earlier the danger of becoming too tactical, and not remembering to kind of bounce back into strategic we can become so obsessed with the numbers that we forget to get out of the building, pick up the phone, call the customer, remember that there might be a completely different reason for why somebody behaves than what we can see through data.
Jason: Totally agree.
Suzanne: I love, also, that you come from a liberal arts background. I come from a liberal arts background as well. I think it's so validating for people who might be listening in who are thinking about coming into product to say that you don't have to be an engineer. You don't have to be a marketer. It helps if you have those skills. Certainly, some of those skills are going to be demanded of you within your role, but if you're creative, and you think this stuff is cool then like come and play with us.
Jason: I totally agree. I think the one caveat I would say is looking back, I think, that I wish I took math a little more seriously than I did because one of the things I realized as I've gone on my journey as a product manager working in the Internet the Internet produces huge amounts of data, right? That's one of the things that has made this into a new industry is that there's many more things you can do with that data. The more comfortable you are with math the more comfortable you are going to be with the data.
I just can't emphasize enough, and now I have my own daughter that as much as I want to foster the creative thinking, and the out-of-the-box thinking, the creativity, the right brain I'm also probably going to be pretty diligent about making sure that she gets comfortable with math, and when she forms an opinion about not liking math it's because she understands it, but just doesn't like it as opposed to me I don't think I liked it because I don't think I really understood it, or how it could be applied. It's only as I've gone through my journey that now I am very comfortable with math, and very comfortable with the numbers, but it took a few aha moments.
Frankly, being surrounded by people in some of those earlier days in Silicon Valley I think that one of the things that's very unique about Silicon Valley compared to let's say now that I'm doing product down here I've done product in Europe is Silicon Valley has an amazing number of the brightest people in the world, right? If you think about it you have Stanford up there. You have Berkeley up there. The top one percent of students in the entire world we're not just talking the states. These are the schools that they go to. Then what they do is they find themselves continuing to stay on in Silicon Valley if they want to work in tech.
I think one of the things that's amazing about the culture is that there's really a culture of everyone's an overachiever, everyone is pretty extraordinarily logical, and I think that that's one of the differences that I've seen between Silicon Valley, and here is that it really is a very unique culture when you take so many people that came from such extraordinary backgrounds, and put them together, but one of the things that I love being outside of Silicon Valley is you're outside of the echo chamber. I think one of my biggest criticisms of Silicon Valley is it's very easy in Silicon Valley to believe that the whole world, or the whole nation, or your whole user base looks like all the people that you interact with day-to-day. That's just not the case. I mean, I remember in 1998 everyone was ordering on Amazon, and everyone was getting the books delivered to their office, but I'm pretty sure that that wasn't happening all across the Midwest at that scale, but it was so easy to believe like this is how everyone does it.
Suzanne: Right, yeah, that's true.
Jason: Yeah, but again, the crystal ball element of that is maybe what they meant to say was this is how everyone will be doing it because I do think that Silicon Valley does happen to be an early adopter market because everyone is kind of savvy, and works in the industry, but it can be easy to get sort of trapped in the box. One of the things that I really love about working outside of Silicon Valley is I feel like I have a bit more perspective on what's really happening in the market at large.
Suzanne: It's like you took the words out of my mouth because I was going to ask having spent so much time up in the valley why L.A. and what's different? Is there anything you've noticed in being down here about kind of the tech landscape because we have our own sort of as you know this blossoming tech community Silicon Beach as some people like to describe. Is there a characteristic to the L.A. tech market that you've kind of perceived?
Jason: Yes. First disclosure I was born and raised in L.A., spent all of my time either in Silicon Valley, or Europe, and then came back to Southern California about 18 months ago, so it's really only in the last 18 months since I've been here that I'm starting to get out into the L.A. community. I've spoken with PMLA, I think it is, and been to a couple of other events. I think that there's some great stuff happening down in L.A. One of the things that I've noticed is it seems to be a little bit more skewed towards media which completely makes sense. If you happen to be selling something, or doing something that involves let's say movie studios, or entertainment brands, certainly, I think that L.A. is much better positioned than probably any other market.