The Power of Community Member Research
The Power of Community Member Research
Time and time again, organizations assume that community member research is duplicative, cumbersome, risky, or incapable of yielding meaningful results. They simply move ahead with their communities and “iterate” as they go.
The results of not performing community member research? Mediocre, copycat brand communities that fail to deliver on employee or customer expectations.
But, brands can avoid this. They can use insights, advice, and qualitative data about community members to build successful, promising communities.
In today's episode, Carrie Melissa Jones shares the importance of undertaking research whether you are preparing to launch your community or are planning a strategic pivot, and how it can impact cross-organizational value.
Carrie Melissa Jones
Alex Mastrianni: Welcome to the Member Engagement Show with Higher Logic, the podcast for association professionals looking to boost retention, gain new members, and deepen member involvement.
Heather McNair: Throughout our show, we'll bring on some experts, talk shop about engagement, and you'll walk away with strategies proven to transform your organization. I'm Heather McNair.
Alex Mastrianni: I'm Alex Mastrianni. And we're happy you're here. Hi, everyone. Welcome back to the Member Engagement Show. Today, I'm excited to share a recent webinar that we hosted, all about community member research. Time and again, organizations assume that community member research is duplicative, cumbersome, risky, or doesn't actually yield meaningful results. They simply move ahead with their communities and iterate as they go. And this might lead to mediocre communities that fail to deliver on expectations, for both the community members and the association. But you can avoid this by using insights, advice, and some qualitative data about community members to help you build successful promising communities. That's what Carrie Melissa Jones is here to talk about today. She'll share the importance of undertaking research, whether you're preparing to launch a community for the first time, or if you're planning a strategic pivot. Carrie is a community leader, entrepreneur, and community management consultant who's been in the online community leadership space since the early- 2000s. Carrie consults with brands to build an optimized community all around the world. She's also the co- author of Building Brand Communities: How Organizations Succeed by Creating Belonging. And it's available at all bookstores. And with that, I will turn it over to Carrie.
Carrie Melissa Jones: Welcome, everyone. So today, what I'm going to be talking about is making the case for community research. So this is really geared toward high- level, maybe not necessarily you're in the trenches doing the research, but trying instead to get the resources that you need to do the research. And if you have a team, helping them understand how long this is going to take, who needs to be involved in this process, as well as what other teams you should involve in community member research. It should never be done in a silo. I've learned that lesson too many times. So we're talking specifically, here today, about talking to community members that are currently in your community or your future community members. So there's many types of research out there, thousands types of research. We're talking here about community member research, so understanding more about who your members are and what their needs are. I'm Carrie Melissa Jones, I'm going to be your guide today. Heath gave you the whole spiel on my bio. In addition to, I work as a consultant. I just want to mention that I am a researcher as well. And so I'm finishing up my masters degree. Working on my master's thesis right now, actually. And everything that I'm sharing with you today is kind of a blend and a mix of industry. This is what academics call what we do, the professional community building and academia. So I know both the academic research process, but I also understand how industry works and what academia sees as ideal is not possible in the industry context. So we have to find what is going to be the best way that we can do this ethically, quickly, but not sloppily, and ground our work in principles that have been taught to researchers for hundreds of years. So that's really... I want to share that I'm not just kind of talking to you about, like run some surveys. This is going to be a lot more serious than that when we're talking about research may be rigorous about it. So today, the clear plan that I will walk you through is, number one, why research matters, why community member research matters, and to whom on your teams. Two, how to plan your research process. So what does this actually take, who needs to be involved. And then three, some tips for analyzing your community research. This is something I get questions about a lot. And I'm going to touch on it at, again, a high level, because so much of research has to be customized to the client and to your needs, but we will talk about at a high level and the big mistakes that I see people making all the time. So welcome. To those of you who have not done community research before, welcome to this journey. And to those of you who have and you're just trying to get a little bit better at what you do, I welcome you as well. A lot of community builders and executives that I work with say that the research process, though it can be hard to get, buy, and to do initially, when they've actually completed it and gone through it, that is when they finally get it. It's when they finally get how community is going to fit into all the other departments in the company, how it's going to fit into the gaps that their customers are currently... The ways that they're not currently delivering on customer experience. And so, really what happens once you've done research is you're not just basing your work on assumptions and things that you've seen other people do, and instead basing it on insights from real people. So this is really when this work can take quite a bit of time sometimes. But once you've done it, it really clarifies everything else. And what I see is people skip this step all the time. There is a kind of this rush nature and people are just trying to iterate instead of research. And to be fair, iterating and launching something, and then watching what people do with it, that is a form of research. We call that maybe, depending on how you're doing it, a quasi experiment. But it will take you so much. They think that if you just skip this step and start iterating that you'll figure it out as you go. Unfortunately, what I see time and time again is that, months from that point, or even years from that point, which is really demoralizing, people will realize, we still have not figured this out. There's something missing here. Something's not working. We're running into roadblocks. We're second- guessing ourselves. We're making assumptions. And they're realizing, we are not our community members. We can't design what they need without talking to them directly. And of course, when you're building a community you might think that you're talking to members all day, every day, but community member research and interviews and things like that go a step deeper. So you're here today because you know better. Then to skip this step. So I want to thank you for being humble and ambitious to be here today because what you're going to be able to do after this is to be able to avoid wasting people's time and also losing the respect, potentially, of your members and losing patients from members who are waiting for you to deliver on the promises that you've made. So what I can say is that your efforts will be far more successful, meaning you'll see more engagement, faster growth in your community, increased ROI, and you won't be exploiting people along the way if you do this right. Okay. So everything else will get clear. You're much more likely to be successful. What I want to share with you and how I think about this is really people who make the case for research are really the research heroes in the organization. Because it's not easy to make the case to step back for a little bit, talk to people, slow down and try to synthesize information, step out of the day- to- day. And this really, if you're going to make the case for this, it makes you this patient leader in the midst of, sometimes, corporate chaos. Okay. So it is no small feat to do this. And I want to let you know that I see you, if you are doing this, and that if you're having trouble with it, I've seen it all. Well, I say that and then I realized that I haven't. So I've seen a lot. Okay. And I can assure you that the struggles you're facing might be unique, but they are not uncommon. So what happens when we don't do this is that we'll see our teams arguing internally, going through potentially like endless decks. I've had a client who just, they kept creating different departments, kept creating different decks, like PowerPoint decks about like, " Here's what the community is going to be. No, here's what it's going to be. No, here." And once the research was actually created they were able to synthesize those decks and throw out all the BS that they had assumed people needed. Unfortunately, they had not talked to users before any of this. So they'd been building all these wireframes and decks without talking to anyone. And so that's a huge waste of time. It wasted months of time. So I want to caveat this with, one, you'll never know everything. Research is going to teach you a lot. It's going to help you understand the motivation of your members, but you will never know everything about everyone and you will never be in a place where you can essentially guarantee success. So research does not guarantee success. I cannot guarantee your success. Unfortunately, any risk that is worth taking in this lifetime, your success cannot be guaranteed. Otherwise, it wouldn't be worth taking the risk, right? So what research does instead is it helps you step back and be able to empathize with people more, which makes you much more likely to create better experiences, write better copy in your community, create more content that actually resonates with people. So your success chances go up, but this cannot guarantee your success. I just want to caveat. Okay. So by the end of this webinar you will have the tools to empower your team to conduct research. And if you don't have a team, that's also completely fine, you will instead understand what's required to conduct research so that you can either work on this yourself in- house, you can hire a team to help you, or you can work with other teams internally to get this work done so that you really can see the big picture. And what I want for each of you to have at the end of this is clarity, clarity on the exact next steps you need to take. So imagine yourself, let's say six months from now, confidently launching something or relaunching something in your community that is completely unique, that has clear workflows that actually work across the business, not just in your community silo, and that you're sure that you can deliver on the promise that you made about what community will bring an organization and what community will bring your members. So you might, at that point, then have a content strategy that mirrors what your customers have asked for from you. You would have like founding members lined up or people really excited to work with you on things. And it might even have content seated and it's going to resonate with people so that your community does not look like a ghost town if you're launching something from scratch. All of that success starts here. It starts with research. It starts with the decision to do something based on your members' needs and not what you've just seen other people doing out there. I get people asking me all the time for tips and tricks. Like, how do I grow my community? What are the hacks that you have for me? I don't have any hacks. What I have for you is a system that will help you to get to know people better so that you don't have to copy paste other people's templates. And this is how we create really unique value propositions that are worth talking about. So that you can stand up on a stage and talk about the efforts that you've done and say, this is the success that we've seen. This is really where it all starts. Research is the key. I didn't want to believe that for a long time, but it's, unfortunately, completely the case. So I hope that you're all ready to do this, ready to jump in. as Adrian said, I'm happy to answer questions at the end of each section. There's three sections to this webinar, as I mentioned. Are there any questions right now, Adrian?
Adrian: No, there aren't any questions, but I just want to mention, we'll be welcome, maybe. We'll have time at the end for questions. And there's ways to get ahold of Carrie as well, later on. Someone who watches this and says, " I have a question." Please go ahead. I'm enraptured to learn about this more because we all know that it takes hard work to have great success.
Carrie Melissa Jones: Yes, indeed. All right. So, first, let's talk about how to make the case across your organization. So why research matters to really every single part of your organization? So there's the really obvious, who needs research? We know that marketing benefits from community research, customer experience and success, support product, R& Ds, sales. All these teams benefit when you do member research, because you're going to learn things about what's broken in customer experience, where people are having issues on the support side. But what you need to figure out is, one, which of these teams are you on? Or are you not on one of these teams? That's totally fine. But if you have each of these teams in your organization, can you confidently say that these teams, each one of them, is in a place to learn and listen from the research? Because what happens when we are not confident of that is that we do this research and then we present the findings and there's massive resistance. Because research often reveals insights that really counter long- held assumptions that these teams might have about your members. So if you're unsure about the willingness for other teams to take in and learn from this research, your first step is to build relationships with people on these teams and figure out what research they've done already, we're going to talk about that in just a second, and figure out how to work with these teams. Okay? So you're going to learn so much through community member research. And you don't want to just throw out all of it, you're going to want to be able to share back these insights with other teams. So here's how you involve these teams, pre- research, if you're not already working closely with them or on one of these teams already. So generally, I customize this flow with each client. But this is just to give you an idea of how this might potentially work. So first you might start out with your marketing or your UX teams. You would, number one, review any existing research that marketing and UX have done. Almost every organization has done some kind of research, whether it's a survey that was run like six months ago and no one did anything with the results, which is just such a shame, or if it was a set of interviews that UX researchers did with some of your early users. Whatever it might be, this is the way that you can start to build that relationship and start to create willingness for these teams, if you're not on them already, to hear and really benefit from the research we're doing. That way, you're not stepping on anyone's toes. You're not duplicating anyone's efforts that, people off more than anything else, when you do something they've already done before. And it's going to help you understand what do we not know today? So we do know this, we don't know this, and this is where I can really be helpful. Okay. So that's the first thing that I generally recommend people do, is go to one of those two teams and start to also gather insights from them. If they already have reports on this, any insights they might have from competitor research, other communities that they might have seen out there, it might be like a Reddit around your brand. And social media as well. So just talking with them about, what have you seen people speaking to each other about? Where do you think there's a need for people to interact many- to- many versus us just putting out messages for them? Then I generally recommend going to your customer experience and/ or support teams. These teams are going to be critical because you are going to learn so much that benefits them in this process. So again, gather insights from them. I actually recommend doing some qualitative interviewing, if you can, of your CX and support team. So talking to them about, say it's a support team and they answer phone calls, what are people calling about most commonly? Who are some of the people that you are assigned to work with and what are their common issues? They also can help you identify customers that you should be talking to. If you don't have a community already and you need to recruit for a qualitative interviews of your members, they're going to help you to identify people that you can then talk further with. So maybe people that they've already built relationships with and they want to deepen those relationships. And in that way, again, you're creating something cooperative. You're not duplicating efforts. And then they'll care a lot more about the findings of your research when you're done. You might also want to go to sales. It kind of depends on your company structure and your business model, but oftentimes sales can give you insights on why you're losing deals as an organization. And also why you're winning deals. And so that can help you to understand where there might be a gap, where people might be concerned that they wouldn't be successful in implementation of your technology product, for instance. And that can also help you. Sales is going to need to know about any community efforts that you launch, because it may or may not be part of the benefits that they sell to customers. So that's going to be really important on the later side of what you're building. And then finally, and again, this is... Sometimes product comes first. It just really depends. You would talk with product and understand where your product team needs more insight and information to build effectively. So again, go to these teams, ask about any research that's been done. Sometimes UX researchers are actually on the product team, that's actually really common, and they have almost certainly done some kind of interviews or brought people in. Well, not these days, but brought them in for Zoom usability testing and things like that. And again, they might have customers that they would like for you to speak with as well. So once you've done all of this kind of internal relationship building, internal research, then you can begin the actual member research. A lot of people, again, skip this entire step and just say, " We're going to run a survey with our members." And you're probably asking questions that marketing is already asking, you're just duplicating a lot of work. So we don't want to do that. Now, I want to kind of break this down for you and how this process might work after you have conducted the research. So don't let this overwhelm you. This is basically the same a diagram as this one, but it's a linear graph here. So marketing, UX, CX, support, product sales, then you do the member research. What I want you to take away from this slide is that once you've conducted the research, what you need to do is share insights back with your sales team. You need to share insights back with product, CX, and support, and marketing, and UX. And how you do that, again, depends on your company culture. It might be that you present in one of their team meetings. It might be that you create a special meeting where you present a deck that shares all the insights and recommendations that you've learned from. You might just send out a deck via email. It depends on kind of how your company works. Maybe you can present an all- hands meeting. Whatever it might be, it's important that after you've conducted this research, that you take what you need and disseminate the information that can benefit these other teams to them so that they can benefit from it as well. Now, I mentioned ops and finance and human resources here because they don't generally need to be involved in the research process at all, but they do need to know what you're going to be doing, how this might affect processes within the organization. So if you end up learning that customer feedback that's coming in through support is not making it to the product teams, then you're going to have to probably create some kind of feedback loop here, right? So your ops team might need to know about that. And they might help you to create this process structure, just really depends on your team culture and things like this. And again what I want to share is that you're likely to face resistance when you're gathering information and when you're sharing the insights back to these teams. Because, as I said, you are most likely going to be breaking some long- held assumptions what people think they know about customers. Sometimes research confirms what's already known, but even when it does that, it can still create resistance because the way people are positioning their problems is often different than the way that you are company is saying the problem. So this is important to know that this will happen. This resistance will happen. That is the work, that is part of the work. So if you're facing it, it's not because you're doing a bad job. It's because you are sharing something that's really shaking things up for them. And that means that you're creating change. And that's an amazing thing to be able to do. So just to kind of summarize and wrap this up. How teams benefit, these teams I've discussed to this point, how they benefit from the research after you've done it. So marketing is going to learn all kinds of new things. Like they're going to gain a lot of new content ideas. Maybe you've talked to people and you want to highlight their stories publicly. Your customer experience teams... You're most likely to learn about some gaps in the customer experience when you do a member research. You're going to hear customers either complain or just give feedback, generally, about that process. And sometimes it's really small things that you learn about. And then you can share that back with CX. They benefit from that. Support, it helps them to create documentation around common problems and things like that. Also if you're creating a support community, it helps your support team to start to prepare for what is to come. And so they can create documentation, start seeding content based on the research that you've done. It also helps them to know that if you are, again, going to launch a support community that their workload should lessen in terms of the reactive work. And they're also going to need to know how their work needs to proactively change as well. So if you're launching a support forum this is like, you're mostly likely on a support team, but this is just such a crucial area. Then you've got product. They're going to gain invaluable advice and insight. They're also going to potentially meet new customers for testing based on the research. And then your sales teams will be able to, from that research, understand where there are new benefits that they could be selling to customers. And they can also begin to share, without concrete details in sales calls, some ideas that have been floating around and just kind of depending on, again, the culture of your company, how open you are about these things. All that kind of depends. But at any rate, all of these teams will benefit from what you're learning. So if you only do the research and only keep it to yourself, then all of this information gets lost and then you don't see the full benefit of research. So the community also, clearly, obviously, benefits from the research itself. So if you're launching a community or relaunching or fixing a community, we've talked about other teams, but it's also the case that doing research is going to help you in all of your work as you're building community. If anyone is asking you and questioning you about why research is going to help you, you can use these bullet points as talking points. I'm not going to read them all out loud. But as I said in the beginning, the bottom line here is that research is going to create consensus. And without consensus you have, oftentimes, just a roadblock where you just can't get things done. That means it's going to save you a ton of time, even if it takes a little bit more on the front end. All right. So that wraps up section one. Can you let me know, Adrian, if there's any questions?
Adrian: Yeah. There are questions.
Carrie Melissa Jones: Okay.
Adrian: Okay. I'm just trying to decide if it's relevant here or more for at the end.
Carrie Melissa Jones: Okay.
Adrian: But I will ask one of the questions. I think the other one we can answer at the end. So the question is from anonymous attendee who's always very famous, which is, how do you build relationships if you haven't involved other teams, i. e marketing during the pre- research stage and currently are executing a community portal?
Carrie Melissa Jones: Yeah. So, if you have not done research up to this point, it's probably going to slow you down, but going to your marketing team at this point and asking what kind of research has been done already is one way to at least start that conversation. My question for you would be like, do they know that you're launching a community portal? Are they aware that this is happening? And if not, that needs to be a conversation that needs to happen as well. I'm sure you have ideas about this, too, Adrian.
Adrian: Yeah. Well, I just... Because I'm stuck on how the, do you build a relationship if you haven't involved other teams? I mean, you start involving them. You go for a coffee or a virtual sync up and you say, " Hey, we're, we're on the same team and we're trying to solve, basically for the same stuff. I'm just coming from a different angle." And overall, as Carrie presented, there's net benefits for your marketing organization. So someone has to take the first step in that conversation. So that's what I'd recommend.
Carrie Melissa Jones: Yeah. One thing I would say about that is make sure that you're not just telling them what you're doing. If you're going to start this conversation, it really needs to be a two- way street. So it's not just, " We're going to start this conversation. Here's this portal I built. Here's how to use it." No, no, no. That's not how you build trust.
Carrie Melissa Jones: You really need to get their insight on what they think about community. So one of the things I mentioned earlier, doing qualitative interviews with support teams, you can also do this with marketing teams. One of the questions I love asking is simply what comes to mind when I say a, say your company is called ABC, when I say ABC community, what comes to mind for you? And just listening to them talk about it and asking up questions about it and seeing if that is in alignment with your thinking. If it's radically different then having that conversation too, but that can literally... You even have coffee for 30 minutes and just talk about that question. That could be one way to start. But it needs to be not just, here's what's happening. I'm going to run you through a demo. No one likes a sales demo out of the blue. Right? So that's a really good question though. I think that's a really common issue. Are there any other questions?
Adrian: There are, but I want to make sure we get through all your slides. So I will...
Carrie Melissa Jones: Okay. Yeah. We're about halfway done. Okay. So planning your community research. Now you've done kind of this relationship building process, you have talked to people, you figured out what research has been done. What do you do next? So I've got this really hand- wavy estimate roadmap here of the general timeline for research and milestones. It can take anywhere from six weeks, I've never seen research to take less than six weeks when it's done well, to, frankly, nine months. I'm working with a client right now where this process is, it took about six months. But what I can tell you is that six- month process. It was not just, we're doing research for six months. It was, we're having conversations for six months. We are finally gaining consensus. We're starting to build things at the same time as we're doing research. So research is not something you have to do 9: 00 to 5:00, all day, every day during your job. But the first milestone is your internal pre- research. So we've already talked about that, but you're also going to be thinking at this point about setting goals and there's really only two goals you should be thinking about. And I will share what those goals are. Your second milestone is then to start to ask questions and make educated guesses about their answer. So in the academic sense, this is about your research question and your hypothesis. So you in a academic social science paper, you always have, what's the question we're trying to answer and what do we think the answer is going to be? And we have to write that down. Otherwise, we can dilute ourselves into thinking that we guess right at the end. Right? so it's really important to ask those questions and then document what we think the answers to those questions are going to be. Then moving into milestone three, you're going to determine the methods to answer those questions. So you don't just say, " At milestone one, okay, we're going to run a survey." No, you figure out what questions you have and then you say, what would be the best way to answer those questions? Maybe it's a survey, but maybe it's qualitative interviews. Maybe it's a workshop. And then at that point you can create a plan for actually executing on that method. Then you actually conduct the research. So, again, this I've seen take anywhere from three weeks to three months. This process, sometimes it's iterative because you realize that your methods not going to work. For instance, I had one client where we had a whole recruitment plan. We were going to run a survey and then do. And we emailed hundreds of customers and we got one response, which is information. It's like, " We're not ready to launch this community. No one's going to show up." So we had to go back and say, " At this point, we're in the conduct research phase, we needed to redetermine our methods." In that case, we ended up doing a set of internal interviews and then getting introduced to people from the support team. So thank you to that support team. And then we moved from the conducting research phase into analyzing for insights and then presenting those insights back. So this is just a really kind of high- level overview of what this looks like. And like I said, sometimes you have to go back and forth and refine as you do this work. So that milestone one is doing that internal pre- research work, which we have already talked about, and confirming your research goals. So these are the two goals I talked about. There's really only two goals you should be thinking about with member research. Number one is you should be trying to better understand one to three things about your members, or better understand them generally. And then number two, your second goal should be to deepen relationships during this process. Unlike doing a academic survey where I'm never going to talk to the respondents again, in fact, I have to anonymize everything, community research and community interviews, especially, are about building those relationships because you're going to continue to work with these people, hopefully. And so both need to be part of what you're creating and building. This is not just a one- and- done kind of a thing. All right. So then we move into milestone two after we've set those bowls. This is where we formulate the questions, our research questions. And then we name the assumptions, so we pre our hypotheses. So at this point, you want to think about what questions do you have broadly? Is there appetite for community? That might be a really broad question. When will be the best time for our programs? What associations do members currently have with the word community? So these might be some really big questions you want to answer through the research. And then you need to write down what you assume the answer will be. If you have done milestone two really rigorously, you will have a good idea of what the answer should be because you have reviewed so much past research that's been done. So what's really important to call out here is that these are not questions that you ask people. You would never ask someone, do you have an appetite for community? Because most people are actually going to say yes to a question that sounds like that and then they won't show up for anything that you do. Because when you say the word community, most people are like, " Ooh, warm, fuzzy things." So these are, instead, we start to break down into like, if our question is there appetite for community, what is the method through which we actually can get the answer to that question? So that's when we move into the actual selecting of our methods based on the big questions that we have. So I have this broken down. I've highlighted qualitative, qual for short, because if you do not have a dedicated survey writer, if you don't have a quantitative research background, quantitative research is, it is an expertise, an area of expertise that is easy to mess up, really, really easy to mess up and get data that is unreliable and not valid. So anything that you learn is basically garbage. It is much harder to mess up qualitative interviews. I've never done an interview where I didn't learn something from someone. And so these are much, much easier to do, especially if you are not working with a team that is experienced in research. So these are often... If you want to gain deeper understanding of a process components, these deeper desires, if people have a desire for community, qualitative interviews are probably going to be your best bet. Again, this is all caveated with, it has to be customized for every client, every situation. Another way to do research that we often don't think about are workshops and doing kind of less focus groups, I am not a fan of focus groups, and more workshops. Because workshops can help us to co- create things. So, maybe if one of our questions, is what programming would people want to be part of in our community? Then we could design a workshop that would help people co- create some of that programming with us. So that's where you might default instead of interviews, because people in interviews, you're not able to ask them, what do you want to do together? They're not able to give you good information. But in a workshop, when they're actually working together, it's going to be a lot more reliable information that you're getting. Then you might also do some competitor research. And this is not just competitors in terms of in your industry, but other communities. So you might want to observe a group that is already active. Again, you could look at like a Reddit in your industry and understand deeply what questions people are asking? How are people interacting? What topics do they want to engage around? So that can help you if you have these broader questions of like, what would this even look like? What would it entail? What else is out there? Then we turn to some more quantitative methodologies. And again, there are many, many quantitative methodologies, I'm just breaking down the ones that are most pertinent to community builders. So first, the most obvious one, are surveys. They're quick, they're easy, they're virtually free. And if you want to understand relationships between variables in your community. So for instance, like showing up to an event and buying a product, then a survey might help. You actually might have that information on your backend of your database. But if you want to understand these two variables and how they interact together, that's where survey research can come in handy. It can also help you with ranking. So understanding not just where people want to talk about, but what are the top three things people want to talk about? So have them pick from a list of 10 in a survey, and then you figure out, here were the top three. That's something that you could not get from qualitative research. And it also can give you really quick pulse insights. It's harder to mess up these short surveys. And then of course, you've got AB testing as well as longitudinal studies. So if you want to understand things like what kind of copy will be successful for us? How will people want to behave over time in our community that we might turn to these other methodologies? Now, teaching every single one of these methodologies... Literally, they teach grad school courses about every single one of these methodologies. So I'm not going to go into depth on this day. This is just for making the case. But no, this is kind of the tip of the iceberg. And I love working on this kind of stuff with my clients and with my students. So if you need to prepare your team for research as you're going into that pre- research phase, you'll want to first spend time creating the criteria and finding the right people to do outreach with. So, for instance, I had a client, they had hired a research agency to do research on if there was an appetite for community. This research agency did not ask among whom. And so they talked to a bunch of VPs instead of talking to, in this case, developers. And so the community that they wanted to build was actually a developer community. And so the research agency went and interviewed a bunch of the wrong people. So you want to be really clear. Like, what are the criteria here? Who do you think will be part of this community? And then how can you recruit from that group of people? Again, if you hire an outside agency, you need to lead the research effort. That horror story I just shared could have been fixed had they led and been a lot more clear with that research agency about who they wanted to talk to. I'm sure that the agency had the best of intentions, there was just a lack of communication there, or a lack of clarity. I also recommend, especially if you're doing quantitative research, but this also works for interviews, do a small sample first. Do a test interview. See how it goes. Have your teammates run through a survey. Have a group of 10 people take the survey and see how it goes. Don't shortchange yourself by just going out and throwing out the research and then not testing it first. In addition, don't leap to assumptions. And this is something important to know for qualitative research, because every interview you're going to learn something and be really excited. So make sure that one anecdote or one interview does not derail you. You need to be looking for bigger picture insights. And then finally stay organized. I can't emphasize this enough. What I do, I'm actually going to share you some share with you how I do this I put things in spreadsheets. I'm not a spreadsheet person, but I know that they can really help myself and my organization skills. And that way I can always return back to the information when I need it in the future. That means not only do you benefit from it now, but you can share the raw data with your teammates. That means that, two years from now, someone else can take a look at your interviews and they could benefit from them. So it's really, really important to stay organized through this process. So we've walked through this kind of linear journey. And again, I've stated this is not always one, two, three, four, five, but we've walked through that full process. So now when we're actually at the stage of analyzing. And this stage should move fairly quickly. You've got all the data, you just need to make some sense of it. This is where we go from here. So are there any questions here, Adrian, that I need to answer?
Adrian: There are a lot of questions, but it's more important to me that you get through the content because I think-
Carrie Melissa Jones: Okay.
Adrian: But take a breath. I'll hit you with two quick questions and then we'll be able to kind of move on very quickly. Why aren't you a fan of focus groups? I think you kind of answered why. And Carolina wants to know what's the ideal length for a survey.
Carrie Melissa Jones: Hmm. Okay. So I'm not a fan of focus groups because I think, oftentimes people are not, they're not going to reveal as much information about themselves. Also building rapport is really difficult. So unless you're an expert in focus groups, it's really hard to get good information from them. Again, like expert survey writers and expert people who run focus groups as experts, they can do magic with these things. But if you are not that person and don't have that person on your team, I highly recommend just going with qualitative interviews because they are so hard to mess up. And then Carolina asked?
Adrian: What's the ideal length for a survey?
Carrie Melissa Jones: There's no ideal length. This all really depends, but the shorter the better. Always, the shorter the better. Hopefully less than five minutes, say.
Adrian: Oh, there's one other question. I know. I know. I'm keeping an eye on the time. Can you give an example of a workshop? Because I came in three times. So we're like...
Carrie Melissa Jones: Oh gosh. Yeah. Well, I... Okay. So before I took my three- week vacation, I ran a series of workshops for a client in which we had five different ideas for programs to run. It was an ambassador program. So we said what, if we run this program, this one, this one, this one, we had people vote on their favorite programs and then get together in groups and say, if we were in this program, what would it look like? And they actually drew out and design things and did funny little skits and stuff to actually show us what they thought these programs would be and what they got. We could hear what they got excited about and what they were sort of like, " Yeah, I'd show up crosstalk." So-
Adrian: Ooh. Yeah. Yeah, maybe the CEO. Woo.
Carrie Melissa Jones: Yeah. So that's one example, but this stuff is highly customized to each client. It really depends. Yeah. Those are really good questions. We'd love to dive in deeper on those at some point. Okay. So let's talk... Oops, go back then. Let's talk really quickly about qualitative analysis. So some tips here. The first thing you're going to want to do is bring all your data together member my member. And I'm going to show you what this actually looks like. Ideally, it's not just you coding and looking at what are all the answers? You have two to three people so that you can check yourself and you're not just making assumptions. Then you can, with those two to three people, conduct a thematic analysis. Again, I'll talk about that. And then you'll separate your findings from your insight. So your findings are, 20% of people said this. That's a finding. An insight is, people do this because they have this motivation. And that is the kind of... Those insights are what product teams like love and what really helps them to design things versus just this kind of raw piece of data. So here's what this process actually literally looks like. I've shared... Some of these are... One of these from grad school, but the other ones are from my actual client work. So here's an example of a raw spreadsheet, your member-by- member insights. The member information goes here. Each question goes up top in each of these columns. The answer goes in each row. Messy and fine. It's all documented. At least all in one place. It's much better than having it in a bunch of different word documents that are not together. Okay. So after you've done all your interviews, for instance, you could start to fill this out. And then you'll move into the actual pulling out the themes work. So generally, how I do this... Again, this is my own kind of neuroses, my own way of doing this work. I do this both for my academic research as well as for industry work. I start to look through each of these answers and I pull out, okay, what is the common theme in that answer? And then I start to group those themes. Each person that is interviewed gets their own color post- it note. And so we get these fun little themes that emerge. If you are less of a tactile learner, you can also do what I've done here in this... This is actually a literature review. But you would have here the name of the person that you interviewed, if it was a qualitative interview, then you would have the themes that you pulled out. And then you would say, okay, did this person touched on this theme or what was their answer? That way you can see these bigger kinds of trends in the data. And then once you pull out the themes, you've organized them, then you'll actually start to bring them together into your findings and your insights. So here's an example of one of the findings that we received from a series of interviews that I did with the client. The question that we had was how should we organize the beta and what content should be included? So we pulled out these themes that we heard in all of the interviews and then we put every... These were not just one person. Some had like five people saying the same thing. We actually included a quote and an anecdote there. So that's something you can then pull from this and put it into a deck. And these are your aha moments that you can share with other people. And that's what people really relish. The other thing here is that when you go back to your hypothesis and you look at, okay, what do we think the answer to this question was going to be? I can tell you, we did not think the answer... Well, first of all, we did not get this detailed. And second, we did not think these were going to be the answers. We thought it was more about relationship building, but what we figured out was like the top need from everyone was help with like onboarding, asking questions, and implementation and configuration. So it's like, we need a support community. We don't need... Everyone doesn't need to be besties here, at least not yet. We've got some other major low- hanging fruit to address. So our hypothesis was wrong. And that, again, that's a great example of, okay, you're hypothesis is wrong, people need to know about this. If we all, based on our pre- research and our assumptions up to this point, we're wrong, then this is a major insight and aha moment. All right. So quantitative analyzing, I'm going to talk about this pretty quickly just so we can move through it and you can ask questions. But just some things to watch out for with quantitative analysis. And here I'm going to talk specifically about surveys. So one is making sure that you have enough data. So what enough data is, it really depends on statistical significance and how many people you have and the variance in the answers. Okay, I'm not going to go into explaining P values and all that to you, but a lot of times I will have clients that will run surveys and they have like 15 responses. And none of that... They're just not going to get enough data to give you anything that's going to be anything more than anecdotal. Sadly, your other teams probably won't care about that. It might be helpful for you, but other teams generally start to question like, " Well, what was the end on that? So how many people did you get to fill out the survey? Second is making sure that you know that surveys can only tell you correlation. They can not tell you causation, because you can't control all these other variables going on. You're not checking before and after. So you're really only seeing how variables are related. Third is sampling bias. So if you only send to one segment of your customer base, then you might be getting data that's accurate to that group but not to the wider group that you're trying to reach. So, again, getting really clear with your team about who you're going to recruit for this research and how you're going to do it is important. Generally, I heard someone say recently that surveys only tell you the opinions of people who like to fill out surveys. So, again, being really careful. Like I once-
Adrian: Was that me?
Carrie Melissa Jones: It wasn't. It wasn't. But-
Adrian: Because I do believe that.
Carrie Melissa Jones: And it's partially true, frankly. I ran a academic study that the survey was 30 minutes long. It took you 30 minutes to complete it. And so our completion was horrific. So really, how reliable is that data? It's just you can't even... It's a long story how you have to justify that. Also be careful of your logical fallacy. So just because one thing is true does not mean that there's only one explanation for it. And then how do you actually mitigate these problems? I've talked about this a little bit already. One is having more than one employee analyze the data. So working with your BI team or whoever else might be on your team, just someone else who is able to look at the data with you and go through it. Again, pre- test anything. Do a pilot of your research instruments. And that's true, not only of surveys, but also interviews. And then one way to do this is to run your research more than once and monitor for changes. So those changes could either be a result of, wow, we've done a really good job in the interim of like we did the survey six months ago, or they could mean that the question you're asking is actually poorly worded. And so therefore is not a valid measure. So running the research than once in a short span of time can tell you if the ways that you are asking the questions is actually going to be valid and reliable. All right. And then briefly we'll talk about delivering research insights. So I don't know how else to get around this. Every organization is like, " We need a PowerPoint. We need a PowerPoint, we need a deck." What does this need to look like in a deck? So you're going to want to present this in a deck. Shareable, if possible, that someone could pass around and people could read and understand without you having to voice over it. And I have samples of these in my online course, actually. But, really what these need to accomplish is getting people excited and understanding, number one, where did this data come from? What is the executive summary and overview? If I only have a minute to look at it, can I look at three slides that just boil it all down for me? Revealing the insights and the findings before jumping into recommendations. So not just saying, we need to do this, but instead saying, X percent of people said this. And we heard this over and over again. And people do this because of this. Therefore, we recommend organizing our content, this is just an example, by doing this. And then finally, after all that's done, beginning to operationalize the work from there. And that's where you'll, when you're sharing this back with all those teams like that diagram I had with all the arrows, this is what you'll share with them. You'll share that deck. You might present to them in a meeting. And then you really need to start talking about, hey, what does this look like in practice? People are saying a product is not hearing support messages, what are we going to do? We need to have this conversation. And you might be the bridge there because you've done this research. So what do you do from here? There was a couple of things you can do. At the beginning, I said if the teams weren't yet ready to listen, you need to start building those relationships. So that's number one. If they are ready to listen, go ahead and talk to them about past research that's been done and start to analyze what they've asked and where there are gaps, in your understanding, to articulate the questions that you need answered before moving forward. You might have 15 questions, but you're going to have to prioritize them. Really, you don't want to have, like five even as kind of a stretch here. Three to four key questions that if you had the answers to them, you'd be able to do things like write- copy more effectively, create content more effectively, just kind of depending on your goals and your state of your community. And then from there, you're going to craft your research plan. So you'll work through that process and actually develop your methods and run the research. So I want to encourage you again, that if you're doing this work, I really do see you as a hero in your organization. You are the patient leader in this world of chaos. And I know that six months from now, you don't want to still be guessing and checking in your community work and just copying what other people are doing and feeling this kind of frantic energy around, am I doing enough. Instead, you want a well- informed plan and a deeper relationships with your customers and your members, a deeper understanding. So this is not easy. This work is really tedious. I do want to also make sure that that's clear, research can be really tedious, but it is worth it. It is always, always, always worth it. Because, as I said in the beginning, research is the way. It's how we get to successful, unique, original communities that are worth talking about and being members of. So this is the work. So welcome. Welcome to this journey.
Alex Mastrianni: I loved hearing Carrie's insights on how powerful community member research can be. Hope you found it valuable, too. If you want to hear more from Carrie, you can follow her on Instagram @ carriemelissajones, or visit her website, carriemelissajoness. com. She's got tons of resources on all things community that you can learn from her course, to her book, Building Brand Communities. And with that, we are going to wrap it up for today's show. Thank you so much for joining us. And we'll see you all again next time.