Introduction It has become common in recent years to gather data on human attitudes and behaviour in building energy research through interviews with building occupants and other relevant actors. We follow this with an overview of the few empirically-based methods that have been put forward to operationalize and measure saturation and identify challenges of applying these approaches to real-life research contexts, particularly those that use inductive thematic analyses. PDF How Many Interviews Are Enough? An Experiment with Data Saturation and Estimates are based on (specified) assumptions, and expectations regarding various elements in a particular study. [9] conducted a stepwise inductive thematic analysis of 60 in-depth interviews among female sex workers in West Africa and discovered that 70% of all 114 identified themes turned up in the first six interviews, and 92% were identified within the first 12 interviews. This is because the population is very varied (from young to old and healthy to sick) and the scope of the research (the experience of accessing healthcare) is quite broad. We chose sets of six because six is a divisor of thirty, and this number was the smallest recommended sample size we identified within the literature. 2005 Nov-Dec;32(6):413-20. doi: 10.1097/00152192-200511000-00014. Our approach includes three primary elements in its calculation and assessment: Base Size, Run Length, and New Information Threshold. Google Scholar. We then identified the number of transcripts needed to meet a new information threshold of 5% or 0%. Of the 114 themes identified in the entire dataset, 80 (70%) turned up in the first six interviews, and 100 themes (92%) were identified within the first 12 interviews (Figure 1). A key advantage is that the metrics are flexible, affording researchers the ability to choose different degrees of rigor by selecting different run lengths and/or new information thresholds. How Many Participants Do I Need? A Guide to Sample Estimation Selecting and interpreting levels of rigor, precision, and confidence is a subjective enterprise. Analytical cookies are used to understand how visitors interact with the website. They found that nearly no new concepts were found after 20 interviews. Apr 21, 2020 2 Image from Unsplash research is conducted to uncover new insights, new observations and find out problems faced by a user group. Data Availability: All relevant data are within the manuscript's Supporting Information files. Qualitative research using interviews is a crucial and established inquiry method in social sciences to ensure that the study outputs represent the researched people and area rather than . Please enable it to take advantage of the complete set of features! The results for Dataset 2 were nearly identical to Dataset 1 (Table 3). Note that in our analyses, successive runs overlap: each set of interviews shifts to the right or forward in time by one event. Not keeping your golden thread front of mind. Methods that calculate saturation based on the proportion of new themes relative to the overall number of themes in a dataset (e.g., Guest et al. But opting out of some of these cookies may affect your browsing experience. As Morse pointed out more than 20 years ago, saturation is an important component of rigor. /* Add your own Mailchimp form style overrides in your site stylesheet or in this style block.
Dataset 2. The concept of saturation was first introduced into the field of qualitative research as theoretical saturation by Glaser and Strauss in their 1967 book The Discovery of Grounded Theory [10]. The 5th and 95th percentiles provide a nonparametric 90% confidence interval for the number of transcripts needed to reach saturation as defined at these new information thresholds. Accessibility (Under our approach, however, a measure of percent saturation as defined by these authors will not be available.). Run length has an effect on the outcome, as one would expect. You can also try to set aside a specific amount of time based on the position you're interviewing for. : expert voices and early career reflections on sampling and cases in qualitative research'. October 31, 2021. How Many Interviews Are Enough to Identify Metathemes in Multisited and If youre still learning new things and generating new codes, you can recruit a few more participants until you reach the point where your themes are complete and little new insight is being discovered with each new interview. For this reason, we have chosen to test 4, 5, and 6 interviews as base sizes from which to calculate the total number of unique themes to be used in the denominator of the saturation ratio. How many qualitative interviews is enough?: expert voices and early It can be done quickly and with no statistical expertise. Unfortunately, there isnt consensus amongst academic researchers on how many interview participants are needed to reach saturation. However, Baker & Edwards (2012) in contributing to the discussion of qualitative. This method also enables researchers to select different levels of the constituent elements in the processi.e., Base Size, Run Length and New Information Thresholdbased on how confident they wish to be that their interpretations and conclusions are based on a dataset that reached thematic saturation. Greg Guest, Arwen Bunce & Laura Johnson. Disclaimer. During the past two decades, scholars have conducted empirical research and developed mathematical/statistical models designed to estimate the likely number of qualitative interviews needed to reach saturation for a given study. Marketing analysts break down data to help guide a company's marketing decisions. By clicking Accept all, you consent to the use of five cookies and the corresponding functions in the website. The bootstrap method is a resampling technique that uses the variability within a sample to estimate the sampling distribution of metrics (in this case saturation metrics) empirically [35]. When modeling the usability issues uncovered from 11 studies, Jakob Nielsen and Tom Landauer found that testing with 5 people uncovered 85% of the usability issues in the interface being tested, and with each additional new participant added to the sample, fewer and fewer new usability issues were noted. Use this catchy poem to remember how many in-depth interviews or focus groups you need. We will prospectively calculate saturation using a base size of 4 interviews and run length of 2 interviews. Careers. Soc Sci Med. Here again the unit of analysis is the data collection event; the items of analysis are unique codes. Saturation is conceptualized as a relative measure. (PDF) How many qualitative interviews is enough - ResearchGate How many interviews are enough? Do qualitative interviews in building N. Denzin and Y. Lincoln, 401-420. Current operationalizations of saturation vary widely in the criteria used to arrive at a binary determination of saturation having been reached or not reached (e.g., Francis et al. The first is that the results are within the range of what we would have expected based on previous empirical studies. The lower the new information thresholdand therefore the more conservative the allowance for recognizing new informationthe more interviews are needed to achieve saturation. Further, from a logistical standpoint, many qualitative researchers do not have the expertise, nor the time required, to perform complicated statistical tests on their datasets. This neutralizes differences in the level of coding granularity among researchers, as the method affects both numerator and denominator. The practical implication of this finding is that researchers can choose a longer run lengthe.g., three interviews (or more)to generate a more conservative assessment of saturation. Calculation is simple. (Thats what a follow-up quantitative study can tell us.) 2021-10-31 18, No. Data saturation is the conceptual yardstick for estimating and assessing qualitative sample sizes. We propose that furnishing researchers with optionsrather than a prescriptive thresholdis a more realistic, transparent and accurate practice. Until relatively recently, researchers and evaluators had to rely on rules of thumb or their personal experiences to estimate how many qualitative data collection events they needed for a study; empirical data to support these sample sizes were virtually non-existent. 2023 Jun 12;14:20406223231173624. doi: 10.1177/20406223231173624. before collecting data! What does "Timely" Mean to Residents? However, in an interview-based study, were looking to understand peoples experiences and needs rather than discover issues in an interface. This blog post is the final in a series of three sampling-focused posts. What Influences Saturation? The inductive thematic analysis included 11 of the 13 questions and generated 93 unique codes. Theres just one small problem with this: saturation, by definition, can be determined only during or after data analysis. BMC Psychiatry. 206-213. Make sure you recruit people who are your target audience and put an emphasis on relevant characteristics that could affect your research questions. Epub 2021 Nov 2. At a run length of three interviews, the median number of interviews required before a drop in new information was observed was seven. Types of Interviews in Research | Guide & Examples - Scribbr Across academic disciplines, and for about the past five decades, the answer to this question has usually revolved around reaching saturation [1, 59]. Field Methods, 18, 59-82. Field Methods, Vol. These new information thresholds can be used as benchmarks similar to how a p-value of <0.05 or <0.01 is used to determine whether enough evidence exists to reject a null hypothesis in statistical analysis. PLOS ONE promises fair, rigorous peer review, No. The process can be applied either prospectively during the data collection and analysis process or retrospectively, after data collection and analysis are complete. Qualitative inquiry and research . 12. To approximate population-level statistics and broaden our validation exercise, we drew empirical bootstrap samples from each of the datasets described above. and How do we know if we have reached saturation? 2006. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc. Interviews involve two or more people, one of whom is the interviewer asking the questions. Using a meta-analytic approach, Galvin [20] reviewed and statistically analyzedusing binomial logic54 qualitative studies. A critical analysis of health system in Nepal; Perspective's based on COVID-19 response. Thus, because theres more variability in the kinds of information were aiming to collect, the point of saturation is often higher than for user tests. This cookie is set by GDPR Cookie Consent plugin. But Creswell (2013) argues for the collection of extensive details about a few sites or individuals. Dataset 3 (Table 4) contained more variation in the sample than the others, which was reflected in a slightly higher median number of interviews and a lower degree of saturation. P30 AI050409/AI/NIAID NIH HHS/United States. How Long Should Interview Answers Be? (Examples and Tips) [26], Fugard & Potts [21], Galvin [20]) ignores the fact that most qualitative research employs non-probabilistic, purposive sampling suited to the nature and objectives of qualitative inquiry [28]. For example, if the hiring manager asks you to describe your strengths, you might speak for 90 seconds to explain where you're proficient. Expert voices and early career reflections on sampling and cases in qualitative research. To validate our method, we use a bootstrapping technique on three existing thematically coded qualitative datasets generated from in-depth interviews. This is not below our 5% threshold, so we continue. 6 likes, 1 comments - CREMES INTERNATIONAL (@cremesinternational) on Instagram: "Quick Research Tips: Interview Skills Recap of Last Week's CREMES Mentorship Program . We acknowledge and agree with these assertions. Though interviewing until saturation was recognized as a best practice, it was not a sufficient description of sample size. Unable to load your collection due to an error, Unable to load your delegates due to an error. Summary:In the early stages of a UX-design project, recruit enough people to gain an in-depth understanding of users experiences and needs. Types of Interviews in Research | Guide & Examples. Saturation is a core guiding principle to determine sample sizes in qualitative research, yet little methodological research exists on parameters that influence saturation. A brief history of saturation and qualitative sample size estimation. Emily Namey, the number of new concepts identified in successive interviews across four datasets. The size of run length effect is smallestvery minimalif employing the 5% new information threshold. The number of new themes in this first run is seven. Figure 2. There's also a school of thought that you should interview 5 or so people per persona. The researchers observed that, of the 36 high-frequency codes (that led to the development of themes) that were added, 34 were added after 6 transcripts and 35 were added after 12 transcripts. The metrics are flexible. eCollection 2023. Code prevalence and timing of code development, by type of code. The advantages of the method we propose are several: Lets consider a step-by-step example of how this process works, using a hypothetical dataset to illustrate the approach. We also use third-party cookies that help us analyze and understand how you use this website. The same can be said for how a researcher chooses to report and interpret statistical findings. */
, We use cookies and other technologies (cookies) to improve our website. Subscribe to the weekly newsletter to get notified about future articles. After a certain number of interviews, there are diminishing returns little new information is learned about the topic of study. 2023 Jun 26. doi: 10.1007/s10508-023-02643-1. Unlike quantitative researchers using statistical analysis methods who have established options for levels of confidence intervals and other metrics to report, there are no agreed-upon metrics to help qualitative researchers interpret the strength of their saturation findings. For Dataset 1 (Table 2), at the 5% new information threshold, the median number of interviews needed to reach a drop-off in new information was consistent across all base sizes. Qualitative research is a type of research that explores and provides deeper insights into real-world problems. Number of new codes identified in batches of six individual interviews (Guest et al., 2006). [23]) are limited by the total number of interviews conducted: the denominator represents the total number of themes in the fully-analyzed dataset and is fixed, while the number of themes in the numerator gets closer to the denominator with every new interview considered, thus eventually reaching 100% saturation. A run can be defined as a set of consecutive events or observations, in this case interviews. Numerous factors are said to be important, including "the quality of data, the scope of the study, the nature of the topic, the amount of useful information obtained from each participant, the use of shadowed data, and the qualitative method and study designed used" (Morse, 2000, p. 1). In this paper we present a way of assessing thematic saturation in inductive analysis of qualitative interviews. Examining 25 in-depth interviews, we found that code saturation was reached at nine interviews, whereby the range of thematic issues was identified. They found that the first five to six interviews produced the majority of new information in the dataset, and that little new information was gained as the sample size approached 20 interviews. FAQ: How Many Interviews Are Too Many? (With Tips) - Indeed Subsequent studies by Francis et al. Base size appears to have almost no effect on the outcome. Fig 1 shows the process, and how base size and run length relate to one another. [22]). Husain MO, Khoso AB, Kiran T, Chaudhry N, Husain MI, Asif M, Ansari M, Rajput AH, Dawood S, Naqvi HA, Nizami AT, Tareen Z, Rumi J, Sherzad S, Khan HA, Bhatia MR, Siddiqui KMS, Zadeh Z, Mehmood N, Talib U, de Oliveira C, Naeem F, Wang W, Voineskos A, Husain N, Foussias G, Chaudhry IB. If our findings hold true in other contexts, it suggests that using a default base size of four interviews is sufficient. Could new themes start emerging later in the data collection process? A smaller company with only 25 employees may want to complete multiple interviews because they're . Students conducting a piece of qualitative research frequently ask 'how many interviews is enough?' Early career researchers and established academics also consider this question when designing research projects. For the next run we add the new themes for the next two interviews, 6 and 7 (note the overlap of interview 6), resulting in a sum of four. FOIA The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. Extrapolating from their data, we see that the first five to six in-depth interviews produced the majority of new data, and approximately 80% to 92% of concepts were identified within the first 10 interviews. PMC They defined the term as the point at which no additional data are being found whereby the [researcher] can develop properties of the category (pg. Options for each metric can be specified prior to analysis or reported after data analysis. Crossref. Galvin R. 2015. Specifically, we address the often-asked question: How many in-depth interviews/focus groups do I need to conduct for my study? Global Health, Population, and Nutrition, FHI 360, Durham, North Carolina, United States of America. Error: Could not authenticate you. We also know from previous studies [9, 16, 29] that most novel information in a qualitative dataset is generated early in the process, and generally follows an asymptotic curve, with a relatively sharp decline in new information occurring after just a small number of data collection/analysis events. But the next question is a purely subjective one: What level of paucity of new information should we accept as indicative of saturation? 61). For Dataset 3, two coders conducted this type of inter-coder reliability assessment on 20% of the interviews (a standard, more efficient approach than double-coding all interviews [2]). This is indicated by a superscript +2 or +3. The cookie is used to store the user consent for the cookies in the category "Other. On average, it takes about five to 10 hours to properly prepare for an interview. Datasets used in bootstrapping analyses. This was qualitative data. e0232076. In this article, Ill highlight some factors that will help you decide. Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet. Schick-Makaroff K, Klarenbach S, Kwon JY, Cohen SR, Czupryn J, Lee L, Pauly R, MacRae JM, Forde B, Sawatzky R. Ther Adv Chronic Dis. Dataset 1 bootstrap distribution: Median number of interviews to reach saturation and degree of saturation at different base sizes and run lengths. The cookies is used to store the user consent for the cookies in the category "Necessary". Timing of code development. For inductive thematic analyses this is a subjective decision that depends on the degree of coding granularity necessary for a particular analytic objective, and how the research team wants to discuss saturation when reporting study findings. And most of us need to justify our sample sizes (to funders, ethics committees, etc.) We hope researchers find this method useful, and that others build on our work by empirically testing the method on different types of datasets drawn from diverse study populations and contexts. Arch Sex Behav. In writing up our saturation assessment then, we would say that using a base size 4 we reached the 5% new information threshold at 6+2 interviews. Transcripts were imported into NVivo [33] to facilitate coding and analysis. As all researchers know, reality often presents surprises. Journal of Building Engineering 1:2-12. doi: 10.1016/j.jobe.2014.12.001. Together they form a unique fingerprint. We used our results to develop parameters that influence saturation, which . on Dataset 1. FHI 360 does not collect any personally identifying information collected through our cookies, although Google Analytics and YouTube may collect your browser information and IP address. Some UX professionals assume that the recommendation to test with 5 users applies to interview-based studies as well. Substantially more women from the Kenya sample were married and living with their partners (63% versus 3%) and were less likely to have completed at least some secondary education. As a rule of thumb, to avoid bias with a single interview, a . Symptoms and coping of patients with dysphagia after anterior cervical 28 hours of interviews for single case studies and 38.8 for multiple case studies. From an applied standpoint this finding is important in that researchers can feel confident that choosing a more stringent new information thresholde.g., 0%will result in a more conservative assessment of saturation, if so desired. There needs to be enough data to get to know the interviewee and yet also enough data about the target sample's perspective on the secondary research . These cookies ensure basic functionalities and security features of the website, anonymously. A marketing analyst is a professional that analyzes data to support a company's marketing efforts. Published on March 10, 2022 by Tegan George. An experienced interviewer can extract in-depth insights from a much smaller number of interview sessions. We would annotate these two extra interviews (indicative of run length) by appending a superscript +2 to the interview number, to indicate a total of eight interviews were completed. This renders a quotient of 11%, still not below our 5% threshold. The cookie is used to store the user consent for the cookies in the category "Analytics".
Dataset 2. The concept of saturation was first introduced into the field of qualitative research as theoretical saturation by Glaser and Strauss in their 1967 book The Discovery of Grounded Theory [10]. The 5th and 95th percentiles provide a nonparametric 90% confidence interval for the number of transcripts needed to reach saturation as defined at these new information thresholds. Accessibility (Under our approach, however, a measure of percent saturation as defined by these authors will not be available.). Run length has an effect on the outcome, as one would expect. You can also try to set aside a specific amount of time based on the position you're interviewing for. : expert voices and early career reflections on sampling and cases in qualitative research'. October 31, 2021. How Many Interviews Are Enough to Identify Metathemes in Multisited and If youre still learning new things and generating new codes, you can recruit a few more participants until you reach the point where your themes are complete and little new insight is being discovered with each new interview. For this reason, we have chosen to test 4, 5, and 6 interviews as base sizes from which to calculate the total number of unique themes to be used in the denominator of the saturation ratio. How many qualitative interviews is enough?: expert voices and early It can be done quickly and with no statistical expertise. Unfortunately, there isnt consensus amongst academic researchers on how many interview participants are needed to reach saturation. However, Baker & Edwards (2012) in contributing to the discussion of qualitative. This method also enables researchers to select different levels of the constituent elements in the processi.e., Base Size, Run Length and New Information Thresholdbased on how confident they wish to be that their interpretations and conclusions are based on a dataset that reached thematic saturation. Greg Guest, Arwen Bunce & Laura Johnson. Disclaimer. During the past two decades, scholars have conducted empirical research and developed mathematical/statistical models designed to estimate the likely number of qualitative interviews needed to reach saturation for a given study. Marketing analysts break down data to help guide a company's marketing decisions. By clicking Accept all, you consent to the use of five cookies and the corresponding functions in the website. The bootstrap method is a resampling technique that uses the variability within a sample to estimate the sampling distribution of metrics (in this case saturation metrics) empirically [35]. When modeling the usability issues uncovered from 11 studies, Jakob Nielsen and Tom Landauer found that testing with 5 people uncovered 85% of the usability issues in the interface being tested, and with each additional new participant added to the sample, fewer and fewer new usability issues were noted. Use this catchy poem to remember how many in-depth interviews or focus groups you need. We will prospectively calculate saturation using a base size of 4 interviews and run length of 2 interviews. Careers. Soc Sci Med. Here again the unit of analysis is the data collection event; the items of analysis are unique codes. Saturation is conceptualized as a relative measure. (PDF) How many qualitative interviews is enough - ResearchGate How many interviews are enough? Do qualitative interviews in building N. Denzin and Y. Lincoln, 401-420. Current operationalizations of saturation vary widely in the criteria used to arrive at a binary determination of saturation having been reached or not reached (e.g., Francis et al. The first is that the results are within the range of what we would have expected based on previous empirical studies. The lower the new information thresholdand therefore the more conservative the allowance for recognizing new informationthe more interviews are needed to achieve saturation. Further, from a logistical standpoint, many qualitative researchers do not have the expertise, nor the time required, to perform complicated statistical tests on their datasets. This neutralizes differences in the level of coding granularity among researchers, as the method affects both numerator and denominator. The practical implication of this finding is that researchers can choose a longer run lengthe.g., three interviews (or more)to generate a more conservative assessment of saturation. Calculation is simple. (Thats what a follow-up quantitative study can tell us.) 2021-10-31 18, No. Data saturation is the conceptual yardstick for estimating and assessing qualitative sample sizes. We propose that furnishing researchers with optionsrather than a prescriptive thresholdis a more realistic, transparent and accurate practice. Until relatively recently, researchers and evaluators had to rely on rules of thumb or their personal experiences to estimate how many qualitative data collection events they needed for a study; empirical data to support these sample sizes were virtually non-existent. 2023 Jun 12;14:20406223231173624. doi: 10.1177/20406223231173624. before collecting data! What does "Timely" Mean to Residents? However, in an interview-based study, were looking to understand peoples experiences and needs rather than discover issues in an interface. This blog post is the final in a series of three sampling-focused posts. What Influences Saturation? The inductive thematic analysis included 11 of the 13 questions and generated 93 unique codes. Theres just one small problem with this: saturation, by definition, can be determined only during or after data analysis. BMC Psychiatry. 206-213. Make sure you recruit people who are your target audience and put an emphasis on relevant characteristics that could affect your research questions. Epub 2021 Nov 2. At a run length of three interviews, the median number of interviews required before a drop in new information was observed was seven. Types of Interviews in Research | Guide & Examples - Scribbr Across academic disciplines, and for about the past five decades, the answer to this question has usually revolved around reaching saturation [1, 59]. Field Methods, 18, 59-82. Field Methods, Vol. These new information thresholds can be used as benchmarks similar to how a p-value of <0.05 or <0.01 is used to determine whether enough evidence exists to reject a null hypothesis in statistical analysis. PLOS ONE promises fair, rigorous peer review, No. The process can be applied either prospectively during the data collection and analysis process or retrospectively, after data collection and analysis are complete. Qualitative inquiry and research . 12. To approximate population-level statistics and broaden our validation exercise, we drew empirical bootstrap samples from each of the datasets described above. and How do we know if we have reached saturation? 2006. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc. Interviews involve two or more people, one of whom is the interviewer asking the questions. Using a meta-analytic approach, Galvin [20] reviewed and statistically analyzedusing binomial logic54 qualitative studies. A critical analysis of health system in Nepal; Perspective's based on COVID-19 response. Thus, because theres more variability in the kinds of information were aiming to collect, the point of saturation is often higher than for user tests. This cookie is set by GDPR Cookie Consent plugin. But Creswell (2013) argues for the collection of extensive details about a few sites or individuals. Dataset 3 (Table 4) contained more variation in the sample than the others, which was reflected in a slightly higher median number of interviews and a lower degree of saturation. P30 AI050409/AI/NIAID NIH HHS/United States. How Long Should Interview Answers Be? (Examples and Tips) [26], Fugard & Potts [21], Galvin [20]) ignores the fact that most qualitative research employs non-probabilistic, purposive sampling suited to the nature and objectives of qualitative inquiry [28]. For example, if the hiring manager asks you to describe your strengths, you might speak for 90 seconds to explain where you're proficient. Expert voices and early career reflections on sampling and cases in qualitative research. To validate our method, we use a bootstrapping technique on three existing thematically coded qualitative datasets generated from in-depth interviews. This is not below our 5% threshold, so we continue. 6 likes, 1 comments - CREMES INTERNATIONAL (@cremesinternational) on Instagram: "Quick Research Tips: Interview Skills Recap of Last Week's CREMES Mentorship Program . We acknowledge and agree with these assertions. Though interviewing until saturation was recognized as a best practice, it was not a sufficient description of sample size. Unable to load your collection due to an error, Unable to load your delegates due to an error. Summary:In the early stages of a UX-design project, recruit enough people to gain an in-depth understanding of users experiences and needs. Types of Interviews in Research | Guide & Examples. Saturation is a core guiding principle to determine sample sizes in qualitative research, yet little methodological research exists on parameters that influence saturation. A brief history of saturation and qualitative sample size estimation. Emily Namey, the number of new concepts identified in successive interviews across four datasets. The size of run length effect is smallestvery minimalif employing the 5% new information threshold. The number of new themes in this first run is seven. Figure 2. There's also a school of thought that you should interview 5 or so people per persona. The researchers observed that, of the 36 high-frequency codes (that led to the development of themes) that were added, 34 were added after 6 transcripts and 35 were added after 12 transcripts. The metrics are flexible. eCollection 2023. Code prevalence and timing of code development, by type of code. The advantages of the method we propose are several: Lets consider a step-by-step example of how this process works, using a hypothetical dataset to illustrate the approach. We also use third-party cookies that help us analyze and understand how you use this website. The same can be said for how a researcher chooses to report and interpret statistical findings. */
, We use cookies and other technologies (cookies) to improve our website. Subscribe to the weekly newsletter to get notified about future articles. After a certain number of interviews, there are diminishing returns little new information is learned about the topic of study. 2023 Jun 26. doi: 10.1007/s10508-023-02643-1. Unlike quantitative researchers using statistical analysis methods who have established options for levels of confidence intervals and other metrics to report, there are no agreed-upon metrics to help qualitative researchers interpret the strength of their saturation findings. For Dataset 1 (Table 2), at the 5% new information threshold, the median number of interviews needed to reach a drop-off in new information was consistent across all base sizes. Qualitative research is a type of research that explores and provides deeper insights into real-world problems. Number of new codes identified in batches of six individual interviews (Guest et al., 2006). [23]) are limited by the total number of interviews conducted: the denominator represents the total number of themes in the fully-analyzed dataset and is fixed, while the number of themes in the numerator gets closer to the denominator with every new interview considered, thus eventually reaching 100% saturation. A run can be defined as a set of consecutive events or observations, in this case interviews. Numerous factors are said to be important, including "the quality of data, the scope of the study, the nature of the topic, the amount of useful information obtained from each participant, the use of shadowed data, and the qualitative method and study designed used" (Morse, 2000, p. 1). In this paper we present a way of assessing thematic saturation in inductive analysis of qualitative interviews. Examining 25 in-depth interviews, we found that code saturation was reached at nine interviews, whereby the range of thematic issues was identified. They found that the first five to six interviews produced the majority of new information in the dataset, and that little new information was gained as the sample size approached 20 interviews. FAQ: How Many Interviews Are Too Many? (With Tips) - Indeed Subsequent studies by Francis et al. Base size appears to have almost no effect on the outcome. Fig 1 shows the process, and how base size and run length relate to one another. [22]). Husain MO, Khoso AB, Kiran T, Chaudhry N, Husain MI, Asif M, Ansari M, Rajput AH, Dawood S, Naqvi HA, Nizami AT, Tareen Z, Rumi J, Sherzad S, Khan HA, Bhatia MR, Siddiqui KMS, Zadeh Z, Mehmood N, Talib U, de Oliveira C, Naeem F, Wang W, Voineskos A, Husain N, Foussias G, Chaudhry IB. If our findings hold true in other contexts, it suggests that using a default base size of four interviews is sufficient. Could new themes start emerging later in the data collection process? A smaller company with only 25 employees may want to complete multiple interviews because they're . Students conducting a piece of qualitative research frequently ask 'how many interviews is enough?' Early career researchers and established academics also consider this question when designing research projects. For the next run we add the new themes for the next two interviews, 6 and 7 (note the overlap of interview 6), resulting in a sum of four. FOIA The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. Extrapolating from their data, we see that the first five to six in-depth interviews produced the majority of new data, and approximately 80% to 92% of concepts were identified within the first 10 interviews. PMC They defined the term as the point at which no additional data are being found whereby the [researcher] can develop properties of the category (pg. Options for each metric can be specified prior to analysis or reported after data analysis. Crossref. Galvin R. 2015. Specifically, we address the often-asked question: How many in-depth interviews/focus groups do I need to conduct for my study? Global Health, Population, and Nutrition, FHI 360, Durham, North Carolina, United States of America. Error: Could not authenticate you. We also know from previous studies [9, 16, 29] that most novel information in a qualitative dataset is generated early in the process, and generally follows an asymptotic curve, with a relatively sharp decline in new information occurring after just a small number of data collection/analysis events. But the next question is a purely subjective one: What level of paucity of new information should we accept as indicative of saturation? 61). For Dataset 3, two coders conducted this type of inter-coder reliability assessment on 20% of the interviews (a standard, more efficient approach than double-coding all interviews [2]). This is indicated by a superscript +2 or +3. The cookie is used to store the user consent for the cookies in the category "Other. On average, it takes about five to 10 hours to properly prepare for an interview. Datasets used in bootstrapping analyses. This was qualitative data. e0232076. In this article, Ill highlight some factors that will help you decide. Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet. Schick-Makaroff K, Klarenbach S, Kwon JY, Cohen SR, Czupryn J, Lee L, Pauly R, MacRae JM, Forde B, Sawatzky R. Ther Adv Chronic Dis. Dataset 1 bootstrap distribution: Median number of interviews to reach saturation and degree of saturation at different base sizes and run lengths. The cookies is used to store the user consent for the cookies in the category "Necessary". Timing of code development. For inductive thematic analyses this is a subjective decision that depends on the degree of coding granularity necessary for a particular analytic objective, and how the research team wants to discuss saturation when reporting study findings. And most of us need to justify our sample sizes (to funders, ethics committees, etc.) We hope researchers find this method useful, and that others build on our work by empirically testing the method on different types of datasets drawn from diverse study populations and contexts. Arch Sex Behav. In writing up our saturation assessment then, we would say that using a base size 4 we reached the 5% new information threshold at 6+2 interviews. Transcripts were imported into NVivo [33] to facilitate coding and analysis. As all researchers know, reality often presents surprises. Journal of Building Engineering 1:2-12. doi: 10.1016/j.jobe.2014.12.001. Together they form a unique fingerprint. We used our results to develop parameters that influence saturation, which . on Dataset 1. FHI 360 does not collect any personally identifying information collected through our cookies, although Google Analytics and YouTube may collect your browser information and IP address. Some UX professionals assume that the recommendation to test with 5 users applies to interview-based studies as well. Substantially more women from the Kenya sample were married and living with their partners (63% versus 3%) and were less likely to have completed at least some secondary education. As a rule of thumb, to avoid bias with a single interview, a . Symptoms and coping of patients with dysphagia after anterior cervical 28 hours of interviews for single case studies and 38.8 for multiple case studies. From an applied standpoint this finding is important in that researchers can feel confident that choosing a more stringent new information thresholde.g., 0%will result in a more conservative assessment of saturation, if so desired. There needs to be enough data to get to know the interviewee and yet also enough data about the target sample's perspective on the secondary research . These cookies ensure basic functionalities and security features of the website, anonymously. A marketing analyst is a professional that analyzes data to support a company's marketing efforts. Published on March 10, 2022 by Tegan George. An experienced interviewer can extract in-depth insights from a much smaller number of interview sessions. We would annotate these two extra interviews (indicative of run length) by appending a superscript +2 to the interview number, to indicate a total of eight interviews were completed. This renders a quotient of 11%, still not below our 5% threshold. The cookie is used to store the user consent for the cookies in the category "Analytics".