Experimentation Framework for Driving Sustainable Growth with Sammy Halabi

Speaker 1: 0:20

[inaudible].

Speaker 2: 0:20

Hey everyone. So, uh, we're really excited. Greg and I for our first episode with a guest of the growth equation. This is one of the most amazing podcasts I've ever heard. And the reason I say that is I was listening to the podcast, I was listening to Sammy talk and I sometimes forgot we were the host. It was just amazing sitting there hearing Sammy talk about some of the most fundamental aspects of a growth equation specifically, uh, things related to experimentation. And we talked a lot about the Toronto tech scene. We talked a lot about how being data informed is significantly better than being just data-driven as an organization. I learned a lot in this conversation and it being our first episode of the growth equation, it was just absolutely awesome.

Speaker 3: 1:03

Yeah, I think just to add onto that was going to be what's going to be great about this episode is like Sammy really brings in a perspective of like how you build out growth teams and how you start that as part of a part of an organization. And I think gross and overused term, we hear a, and I think will really help people understand like how you build teams and then how do you experiment in the right way and what is the, what is the right mindset and framework can you do as an organization to be successful within building scalable growth?

Speaker 2: 1:28

And one of the most exciting things is that wave was recently acquired and one of Canada's largest tech exits, uh, by H and R block. Uh, so we also talked a little bit about that and myself as a user of wave, uh, it's, it was just amazing to hear Sammy, so we're sure you're going to get the same kind of value and same kind of insight that we did. Let's dive right into our conversation with Sammy. Awesome. Let's get started.

Speaker 4: 1:55

[inaudible]

Speaker 5: 2:00

Hey Sammy. So welcome to the first very first episode, the growth equation podcast. Thanks. Or thanks for coming in.

Speaker 6: 2:06

I am so stoked to be the first one.

Speaker 5: 2:08

Amazing. Amazing. I think just a great way to, to kind of, um, come on the podcast is maybe just kinda kick it off by just telling us kind of maybe who you are and some of some of your background. And then from there we can kind of have a really interesting conversation about growth.

Speaker 6: 2:23

Yeah, sure. So I'm a growth manager at wave wave is a, uh, we really help small business owners simplify their financial lives. So basically anything from keeping track of their expenses, doing their bookkeeping, paying their employees, sending invoices, getting paid, all the fun stuff that maybe they don't know how to do right off the bat. Uh, we help them with that. So I've been doing growth at wave for about six months now, about six months now. Uh, prior to that I was a product marketer, um, both at wave and another tech company in Toronto. And before that I was in Corp dev and my background is not in growth at all. So like most people in growth, I, uh, I never went to school for it. So kind of evolved into a, over my career. One question I have is, you know, I think we consistently always hear, especially within the tech startup theme, like the word growth, it's probably an over and overused term, right? And I think what, what really is like, what does I grow up in, in your day? I mean, uh, that's a, that's a great question. And I've been trying to explain it to my mom for awhile and I still can't, so I'll, I'll do my best on the podcast. Uh, what does the growth manager do? And I guess what is growth? Right? So, so Andrew Chen, uh, he says it's the scientific method applied to KPI's, which I thought it was kind of a cool definition. And I mean, you've heard all the terms, right? Like growth hacking is at the same as product lead growth is a product management like it should have. We all be doing growth. Uh, I hear that a lot. So the way I think about it is it's kinda like the art and science behind how companies grow, right? So it's building and testing what I call or what some people call growth models through rigorous experimentation. It's thinking inside and outside of the product. So it's not just, um, product management. It's not just product focused. It could also be acquisition focused and it's doing it in a way where you're always kind of thinking about a core metric. Um, something that's kind of like a North star for you and trying to move that. But otherwise, what does a growth manager do? So it can take on many different flavors. Sometimes my job is similar to that of a, of a product manager or PM where I'm working on either kind of designing or managing a pipeline of experiments, defining and prioritizing what that pipeline should look like and what we need to run next. And it's trying to have like a holistic view of what a business is and what it could be. Um, and then figuring out, all right, like what are the things that we need to do to get there? So that's sort of like what I do on a day to day level. I sit within a, a a growth team. So we're like a pod where there's me, a few engineers, a growth designer and a growth that analyst. So I'll sort of, uh, how we're set up and then just just on that, like it almost sounds like you are like this growth experiment Taishan scientist is that if that's fair. Yeah, that's fair. I'm like there is a science behind it. You know, we, we throw it all those terms like statistical significance and sample size. Like w we use that language every day, but I also don't want to obscure it too much. So the problem with calling us growth scientists is you almost lose the, uh, the humanity behind that. So a good growth manager and a good growth team, uh, they are what I like to call data informed, not data-driven. So it's really finding that fine balance between, you know, being rigorous, being data kind of focused and obsessed, but also being customer focused and obsessed. Sammy. Uh, I'd be curious to find out what kind of tools do you use? It's kind of on a day to day basis. Are they in house tools that you use to kind of gather that business intelligence and that growth information that, that guides your decisions or are they off the shelf tools that you use? How do you actually accumulate that data and make informed decisions based on it? Yeah, that's a great question. Uh, and for us, so wave is like a 200 plus person business now. Um, we have kind of a built-out data infrastructure and data team, so it's a mix of in house and off the shelf tooling. Right. On the data side, a lot of that is just being pulled from our warehouse and um, you know, the data team's pretty pretty on top of making sure that infrastructure is up to date. But on the experimentation side, we'll use things like optimize li to actually run the AB test. Um, tools like that. Sam starting to become a growth manager first time in a fast growing start up. Um, how do you, what's your framework like on just like deciding what experiments to run or like how do you even, what's the like, what's that approach? Even like I'm, where are you, where do you even begin that? That's a great question and I don't want to cop out and say it's different at every business, but it is really different at every business. I would say step one is like before you even decide or figure out like what experiments you want to run, actually make sure you understand your funnel, right? Like how does your business actually grow? How does rev, how's the revenue actually being generated? Um, what metrics do and do not matter. So like the first thing you should do is just make sure you are rock solid on that. So w when we set up growth at wave, that's, that was sort of our approach, right? Like what is our view of the growth model here? So like what is the one metric that sort of matters to us? If we were at Uber, we probably would have said, um, increasing the number of brides that people take. Right? Something like that. And then figure out what are all the input metrics to that, right? So, uh, taking, let's take Amazon as an example. Um, Amazon grows by having more people transact on it, right? Like more orders. So if you were thinking about like input metrics to Amazon, you'd be like, all right, there have to be different categories of products, right? Amazon will grow more if it's not just books, it's also TVs and electronic equipment, right? And the inventory for each of those verticals. So like figuring out what those input metrics are and then go through the process of figuring out experimentation. If we want to talk about like designing an experiment, prioritizing those. We kind of go through four phases for designing an experiment. The first phase is what we call explore. So that's very much, I'm trying to figure out what we call a problem statement, right? Like, Hey, what is this big hairy meaty problem, uh, in our business today? Something like, I don't know, uh, if I was Uber, only 30% of people that download the app ever actually take a ride. Just a very large problem. From there you move into a design phase where you figure out what are the different ways in which I could solve for that. And in solving for that, you try to make sure that you get your feasibility done. Like engineer's tell you that this is not possible. You try to figure out what metrics you would move. You try to understand like is this a high dependency or low dependency project? What sort of investment would be required here? What's the hypothesis here and is the impact that it would have big enough to justify this? And that's how you sort of make a decision on whether or not you would go forward with a project and how to rank them against each other. And then you move into your execution analysis phase and the execution really should be a very small part. A lot of the work should actually be done upfront and figuring out what you want to work on. Uh, and then the most important one for me is actually the analysis. Because if you haven't learned something by the end of it when or fail, then you have done a good job. So making sure that you feed that learning back either into a new experiment or hopefully into shipping in the product.

Speaker 5: 10:42

Amazing. Amazing that, that, that totally makes sense. And I think just one question, um, that you were talking through just talking about where you first started off and talking about metrics. I think it was interesting how you, you started talking about almost like call that what that, what that user needs to do, right? Because when people say metrics as an example, people tend to, their mind tends to be like, Oh my metrics going to be like more customers or more revenue as an example. Right? But you went specifically in using the Uber example as like they want to give more rides. So maybe you could just talk a little bit about metrics and is it about focusing on an output metric or an input metric? Is it about focusing more on like something that drives product value versus like I'm focusing more on like call it business metrics that customers or revenue. Like how do you, how do you really define what, what is the right metrics to be measuring?

Speaker 6: 11:37

That's a, uh, that's a very good question. It's a really hard question to, to answer in a few minutes, but when you're picking your like one metric that matters, right? That big large metric, that metric, it should be like almost directly correlated and like causally related to revenue, right? In my opinion. So if the number of rides go up, then Uber's revenue will go up, right? It's just, it's simple math, right? It's arithmetic at that point. But when you're focusing on like what to work on, it's important to understand that that kind of North star metric is very, very far away from where your experiment is, right? So generally speaking, when you're running an experiment, you actually want to target those implement tricks, the things that sort of feed into that North star. Um, because there are a lot closer to your experiment. So like I can give you, um, kind of like a tactical example. Let's say you want it to, um, improve checkout on an eCommerce website, right? You would probably run that experiment right on the checkout page. You wouldn't necessarily run that like five pages prior, right? Because it's much more directly related to what it is you're trying to move. So similarly, when you're thinking about moving the number of rides, you're probably gonna focus on one part of their implementer X, like the booking process or you're going to focus specifically on how they invite their friends. Right? So that's sort of my approach and our approach here at wave two that we, we try to make sure that we're not targeting something that's too far away from where our experiment is. Does that, does that make sense? Absolutely. Uh, if we take a step back though, obviously growth is not something you necessarily learn in school. It's a very important aspect of running, especially in a small company that requires growth to, to even survive and at a, at an accelerated rate being, you know, waves first growth manager and jumping into that role only a couple of months ago. W, what was your introduction to that role and how was it defined and what were some of your, you know, first key responsibilities that you had to take over to start succeeding at it and making it something that you want to build a career in? Ooh, that's a, that's a tough question. Um, I was lucky. Um, I was the first growth manager, but, uh, some of the, um, core members of the team, uh, like my manager had been doing it for a while. So I was lucky coming into this role because I had someone to learn from who had done it before. I'd say the most important things are attitude, right? Like coming into it with the right mindset. I would say if you want to be successful in growth, you need to make sure, um, that you're not approaching thing. You're approaching things from a, um, from a critical lens or through a critical lens, not taking for granted things that may be seen as common knowledge within your business. So making sure that you're always questioning the way that we've done things prior, right? Not because they're wrong, just because it's important to question. Um, I read and flee like there are some great resources out there, um, for growth specifically guys like, uh, like Jeff Chang runs a thing called growth edge blog, which was great cause it really gives you the ins and outs of like all the different ways, different growth teams are set up experiments. Uh, he's run before. So you kind of get a sense of um, of how it's being done in the industry. And there are a bunch of more resources like that. But then I'd say the most important thing for me personally was actually just trying and failing at our first couple of experiments. So we intentionally, we intentionally ran the first two or three experiments without a lot of structure and a lot of process because we wanted to understand how we as a team, we're strong and weak, right? Like how should a growth manager at wave interact with a growth engineer at wave and interact with the growth designer at wave new growth data analysts that way. So there's a lot of baptism by fire. So I think that's important for any new growth team coming together and just getting a very strong sense of your business going through the product and understanding all the different things that are weird and just writing them down somewhere. I think that's one of the most important things that you can do when you first come into a role like this. You know, it sounds like growth, at least your role is focused a lot on understanding the product and perhaps making tweaks to the, to the product and the way people interact with it. Is that kind of capturing your role properly or it can grow, I imagine extend beyond just tweaking the product and all of that. You really need to think about, um, growth experiments in breaking up into two categories. One being innovation and the other being optimization. Generally speaking, the larger your business, uh, the more established your funnel, the more traffic you have, the more you can operate on the optimization side, right? That's why I'm a growth team at Google. Could run, experiment and have like 40,000 shades of blue being tested, right? Because they only need to see a small lift and they have enough sample to run an experiment like that. The smaller your business or the smaller your funnel, the more your experiments are going to focus on, um, whole skill innovations, right? So instead of like testing how many shades of blue, you might remove the budding completely or you might just totally change that experience because you need to make bolder changes to actually detect a difference. Right. So I think that's the other aspect of, um, growth experiments.

Speaker 5: 17:33

And just on that, like, is it about like just talking about optimization versus innovation? Like is it about doing like 70% innovation and 30% optimization? Do you think about experiments or is it like a balance of both or the question or is it, it depends based

Speaker 6: 17:50

on your business. It definitely depends based on your business. Like, if you're just starting out and you know your funnel is like a few thousand large or a few hundred large or whatever's relevant to your, uh, to your business, then you're probably going to focus a lot more on the innovation side because you're really almost testing for product market fit at that point, right? Whereas as you get larger, you can afford to run these optimizations and mix in innovation experiments or innovation styles, experiments, uh, in more infrequently. So it really depends. I think it's fundamentally important that you don't fall into the trap though of only running optimization experiments and only running them early because it's almost like the difference between what Matthew would call like a local maximum and a global maximum, right? You can optimize for a local maximum, but you might be missing the whole big picture. So it's really important to do both.

Speaker 5: 18:51

Can you talk a little bit about magic moments? Cause I think that's a fascinating concept. Uh, I think it's a great way to get buy in from customers and our are your experiments around how to optimize and and those magic moments that somebody at wave or at any other organization has installed or you know, are you innovating and creating brand new ones? Can you talk to us maybe a little bit for those who don't necessarily know about the concept of magic moments, dive a little deeper into that and then perhaps how it manifests itself in a, in a growth related room.

Speaker 6: 19:25

Sure. Um, so magic moment at it's like core is can I generate an emotional response? Right? Can I make you feel good or happy, um, when you've achieved or done something right? So I wouldn't say that our experiments necessarily try to move like number of magic moments, right? But part of our framework for designing an experiment, the, the last part, so we call it the hype framework, like hypothesis, investment precedent. The last part of that is experience, right? And basically we ask ourselves when we're designing experiment, like, is this change a good user experience? It's like this kind of sense check, right? Is this experience going to be something that would generate an emotional response? And so when I think of magic moments, I don't think there's one. I think there's a ton. I mean there's obviously like a core one early that helps people retain into your product, but you should be creating like many magic moments throughout the experience, right? And so I always try to make sure that our solution has a little bit of that in there. But if you're looking for a sustainable growth, you need to couple that with like a product experience or core product value, like something that actually solves a real problem. So like a great example is Uber, right? Uber is, I don't know if you guys remember the first time you booked an Uber, but I distinctly remember it because a friend sent me a link. Uh, I was in London at the time and I download this app. I have no idea what it is. I have a 10 pounds voucher already preloaded in it. I book a ride, some car shows up and that never happened before. Right? Like until then, I'd always been hailing taxi off the street and I get in and then I go home right after a long night out. And that was magical. Right. But it also solved a real problem. It got me home. So it's when you get those two together, that's when it really starts to degenerate. That flywheel, right.

Speaker 5: 21:30

Yeah. That Uber example resonates cause um, you sometimes forget, you know, when you're talking about, do you remember your first Uber ride? I don't, but I remember when I wasn't able to request an Uber ride for the first time when I landed at an airport in the city where they didn't have Uber. Les and I pull out my phone and I see nothing there that was striking and probably the exact same capacity where you fundamentally changed behavior to a point where you just don't remember life otherwise. So I mean you, you talked about something that I thought was really interesting cause I've always used the term data-driven. I wanted to be more data-driven. Uh, I want to find more ways to gather more data. Can you talk a little bit more about what you mean by being data informed as opposed to data-driven and what are their sources of data or other opinions or you know, even gut feelings factor into you actually determining whether an experiment is successful and how you can actually factor that into making a larger decision for the organization or for the product or anything like that? Yeah, absolutely. So the difference to me, if you're

Speaker 6: 22:31

purely data-driven, you're probably running a bunch of query, pulling a bunch of data and sort of developing this sort of um, is data-driven view of your funnel, right? Like this many people do this, this many people drop off and this many people drop off here in the funnel and this is what we've got. Right? And that gives you part of the picture, right? It, it tells you what your funnel looks like or what your business sort of looks like from a quantitative perspective. But it doesn't necessarily tell you why it is that way, right? Like why are people dropping off here or there? And so being data informed is making sure, like, have I actually spoken to a customer, um, who's in this part of the funnel and that this is happening to have I done any customer development interviews? Have I tried to understand the psychology of the customer at this point? Because what ultimately you're also going to be designing a solution for them, right?

Speaker 5: 23:28

So being data informed to me

Speaker 6: 23:31

is making sure that you actually have your finger on you actually remember that you're not designing an experiment for the sake of an experiment. You're actually designing it for a person, for human being. Um, because ultimately if you're not doing that, you're, you're going to miss the Mark. More often than not. There's a lot of different answers to a problem or a lot of different solutions to a problem. Um, and there's a lot of different ways to interpret a problem. So being purely data-driven, you might miss that sometimes. So, with,

Speaker 5: 24:06

with that being said, one, one thing that, you know, you've kind of sparked a thought on my end is, you know, how do you think you, you know, especially for all of those, you know, those fast growing companies we have right now within the Canadian startup ecosystem right now, it's like, how do you create a culture of experimentation? It's a mindset shift in a lot of ways. Like as you were been talking about is like you have to focus on one metric that matters. You have to focus on thinking about learning over, just winning, like experiments, winning all the time. Like what do you think are the important thing you need to instill in a mindset within an organization in order for growth teams to be successful?

Speaker 6: 24:40

I think there's two parts to that question. So the first part is like how do you create a culture of experimentation or like a growth mindset of the business? And that kind of varies depending on like where is leadership rooted in that business, right? Like has it traditionally been a sales letter organization or product labor organization? Um, just understand the dynamics around that because usually that'll give you a sense of, um, how the business views change, right? So they're the great case. Um, Apple didn't run experiments for a very long time because they were so design-centric that they were obsessive about maintaining a certain user experience, right? So they were resistant to, um, certain types of experiments because it just wasn't in their DNA. Right? But another business might be the complete opposite, right? Uber from the start has been about we need to test everything, right? Like every person has a different version of the app and we're serving them different things at all times. So understand the culture of your business to understand where and when and how much you can push. The other thing I would say is that when you're building a growth team, if you want to have that sort of growth culture or mindset, one is making sure that you staff it with people that are naturally curious, um, people that are pretty critical. Making sure that all voices are heard, that everything is questioned and nothing is taken as given. Um, so that that becomes a, uh, that's almost a, a, a people and culture thing.

Speaker 5: 26:19

What is this like one piece of advice you would give companies that really want to focus on growth and want to build out a growth team?

Speaker 6: 26:27

One is like how willing are you to actually run experiments, right? How much are you willing to question and be critical about? Sometimes a experimentation can feel uncomfortable, right? You're challenging conventional wisdom. You're not sure how your customers are going react. Um, maybe they're going to hate something, right? So are you, are you on board with that? That's the, my first piece of advice is just make sure there's a bit of soul searching there that you're ready for it. And then the other one is just, um, don't expect growth to be a silver bullet. Right? It's very rare and I'd say almost impossible to find that one thing that's going to turn you from a, a, you know, 10 million ARR business to a$1 billion ARR business in know, in a year. Right? It just doesn't happen. Think of growth. More like, am I going to find a series of small wins and then a big win every once in awhile. Right? So just having the right expectations, uh, given that we're all from Toronto, uh, I just be curious to hear your thoughts guys on how you think the Toronto or larger Canadian ecosystem startup ecosystem, small business ecosystem does, uh, in terms of understanding growth, implementing fast experimentation strategies and, and just, you know, the overall notion of growth overall. I think definitely one thing that differentiates us from say our, our, our partners down South, especially in the Valley, is that companies seem to grow and exit and all a lot faster there. Is it just a cultural difference? Do you think people play it a little bit safer here? How can we run experiments and have that fast paced growth mindset here? That's a great question. Uh, I love the Toronto tech scene because we are a little bit different, right? Uh, and I like that it's, it is a little bit uniquely Canadian, right? So that, that's a good thing and I wouldn't want us to lose that. I would say that when I see, or when I look at the growth landscape in Toronto, it's sparse, right? Like there aren't a lot of teams set up in the way a, a growth team at Dropbox would be set up for Uber. We can set up, part of that is because, um, there haven't been a lot of businesses like that in Canada where those sort of teams kind of diffuse into the ecosystem. And part of it is that, you know, growth as a mindset and as a function started somewhere else and we're just starting to, uh, to import those ideas, right. And that way of thinking. So yeah, for sure. Uh, we have a little bit of catch up to do and, um, I think the way to overcome it is just to make sure, like, are we constantly looking at what's out there? Are we looking at how other teams are scaling and building and staffing and are we picking and choosing what we think is relevant to businesses interim, right. In businesses like ours and to the, um, I guess the skills that we have in our market. So, yeah, there, there's definitely a difference. I see the gap closing, but it's still there. Yeah, the Toronto

Speaker 5: 29:42

ecosystem zone and like a very early stage right now. And the biggest thing is our talent is, is just growing. Um, and we're importing ideas and ultimately what needs to happen is all of those, those bigger type companies and you're starting to see of the Shopify is of the world, et cetera. Ultimately what needs to happen is you need to have those types of companies need to be successful and they need to take those ideas and more important with people they need to go down into to help those more earlier stage companies help, help grow. I think the, um, one of the biggest challenges in specifically in the Canadian specific thing is that we don't think big enough a lot of times, um, and we always try to optimize the risk, right? Like you see a lot of different, especially within the Toronto ecosystem here, a lot of like beauty. SAS is a big thing, right? And that tends to be like a sweet spot. And that tends to be something where you're just grew up minding your risk, right? You're, you're building solid revenue compared to like B to C companies that you see a ton of influx of cash, right? And you're trying to scale it like a rapid rate to get as many, many users as possible and then their own, then they're thinking about their monetization model maybe a few years down the rails, right? Like the whole, uh, you know, Facebook model is a great example of that. Right? Same with same with like snapshot, right? So I think to me personally, I feel like it just, it's going to take time for that, for that mindset to grow, but we won't become those types of places because I think it's, there's a, there's a way to be distinctly Canadian, but that's not a bad thing. Well, I also think, I mean there've been a lot of successful and massive exits in the Valley and down South and that allows companies to feedback and founders feedback into that ecosystem, invest in new companies and new initiatives. And I think, you know, even just waste the acquisition with H and R block and all of that, I mean, so many new and exciting, you know, you know, one of my favorite years ritual, they are on a dedicated and fast track to just something massive. And I think over over the coming years we'll just have that feedback loop and, and that, that just, that snowball effect we're gonna see a lot more I think. So I'm really optimistic about it and I also do really like the whole Canadian startup environment. It's definitely unique and I'm greatly valued. While I'm semi, I think we'd love to, you know, cap off this episode and ask if you had any final thoughts, uh, around growth. Anything you didn't get a chance to really talk about. Um, but we really appreciate your time.

Speaker 6: 32:01

Uh, yeah, this was a lot of fun. Um, and I'm really excited to see the other speakers you guys bring onto this cause I think this is really needed, especially in the Canadian ecosystem, but just more broadly and more people talking about growth I think is a good thing. And, you know, I'm excited to see great growth teams popping up everywhere. Right. Um, but yeah. Thank you very much guys.

Speaker 5: 32:23

Yeah, and just on a, like a, a last, last ask for you is, uh, just for anyone listening, uh, do you have any sort of recommendations on any sort of like additional resources to read online? Uh, and then how do, how do people find you? So, uh,

Speaker 6: 32:37

read growth edge blog religiously and I follow Andy Johns, uh, the guys at Reforge sprint Balfour, also talking about great things. Uh, so there are, there are definitely good out resources out there. Uh, and yeah, if anyone wants to get in touch, they find me on LinkedIn or on Twitter at semi Halabi. And happy to take any questions. Yeah. Thank you so much, Tammy. That was an amazing episode. I, it was our first one with a guest online,

Speaker 5: 33:04

so, uh, uh, it was a real privilege in that I personally learned a lot. I'm sure you to Greg as well, and we'll definitely look forward to the next one. So thanks again.

Speaker 4: 33:41

[inaudible].