The Method to Our Madness

Can AI really handle coding qualitative data, or is it just a shortcut that costs you in the end? In this episode, Stacey and Bill sit down with Dr. Asher Beckwitt, a researcher with 26 years of experience across 5,000 qualitative and mixed-methods studies, to unpack where AI genuinely helps with qualitative data analysis, and where it falls short. Asher walks through choosing compliant, privacy-safe AI tools, why human verification of AI outputs is non-negotiable, and how to document AI use for transparency and IRB compliance (including the often-overlooked step of fully deleting data from AI systems). They also break down the differences between grounded theory, thematic analysis, and phenomenology, and where AI fits (or doesn't) in each. A must-listen for researchers weighing AI tools against methodological rigor in their qualitative research projects. This episode also touches a fish consumption study Asher led for a tribal nation using community-based participatory research methods. This was a separate, non-AI project worth a listen on its own. Dr. Asher Beckwitt can be reached at asherbeckwitt@gmail.com or www.asherconsult.com . A summary of this episode can be found here . The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

In this episode, Stacey and Bill talk with Joan Wagner about how systems engineering and industrial engineering (IE) work together. The former includes defining and verifying requirements, while the later involves optimizing processes and efficiency. Highlights include: The "Systems Engineering V" model and the importance of setting clear project boundaries Why Lean and Kaizen work best when the people doing the work drive the improvements A story about building stakeholder buy-in with a resistant shop-floor manager Costly planning failures, including a machine built too tall for its doorway Where Six Sigma works well, and where it gets misapplied The ongoing search for better systems-engineering documentation tools Joan's outreach work with ISE Day , bringing IE education to K–12 students Episode notes can be found here . The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

Focusing on policy recommendations as part of the evaluation is critical because health data without policy recommendations grounded in data creates a dangerous blind spot for policymakers, often leading to well-intentioned regulations that fail on the ground. While quantitative metrics can pinpoint a drop in Medicaid enrollment, for example, they cannot capture the confusing administrative hurdles, conflicting agency rules, or frontline operational friction causing those trends. By integrating targeted focus groups and interviews into the data findings, evaluators can diagnose the actual root causes of systemic failures which can then provide opportunities for strong policy recommendations and policy frameworks. In this episode, we're joined by Nalini Pande, Managing Director at Sappho Health Strategies , to talk about the unique challenges of evaluating healthcare programs and translating results into policy. Drawing on her deep background in Medicaid, Medicare, and ACA policy, Nalini makes the case for weaving policy considerations into an evaluation from day one, rather than adding them on at the end. A summary of this episode can be found here . The music for this podcast was written and produced by Stacey Merola. The artwork was created by Zoe Targino.

What do mixed methods interrupted time series designs, business development, and moose have in common? More than you might think. In this episode of The Method to Our Madness, we explore the power of mixed methods interrupted time series designs and how they help researchers and evaluators understand change over time. We also share insights from the world of business development, including the challenges and opportunities of building partnerships, pursuing new work, and growing a business. Along the way, we take an unexpected detour into the world of moose, because sometimes the most memorable conversations happen when data, strategy, and wildlife collide. Whether you're interested in research methods, professional growth, or simply enjoy hearing smart people wander off on a tangent, this episode offers practical insights, candid reflections, and a few moose stories you didn't know you needed. Podcast music written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino.

What if your logic model is too simple to capture how your program actually works? Hosts Dr. Stacey Merola, and Bill Pate sit down with Dr. Bernadette Wright to explore knowledge mapping. A knowledge map is a visual approach that uses circles and arrows to show the interconnected web of causes and effects behind real-world outcomes, instead of forcing everything into a rigid input-output framework. Dr. Wright walks through how knowledge maps are built (often with 20-25+ interconnected concepts), why multiple reviewers matter, and how the process helps teams surface feedback loops, catch oversimplified solutions, and get aligned on what's actually driving results. The conversation also covers popular mapping tools, how the approach compares to traditional logic models, and where generative AI might fit into the future of concept mapping. A must-listen for evaluators, researchers, and program leaders who want a clearer picture of complexity. An example of a knowledge map can be found here. Source: Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge A summary of the episode can be found here . The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

Dr. Stacey Merola and Bill Pate sit down with Dr. Toks Fashola, research professor at American University, to unpack what it really means to do culturally responsive program evaluation. Dr. Fashola breaks down how spurious variables can quietly become confounding ones by using everyday examples like parking tickets in Baltimore and medication adherence in diabetes care and explains why understanding a community's history and context is essential to getting evaluation right. Drawing on her own experience with an after-school program in Baltimore, she makes the case for treating stakeholders as co-designers rather than subjects, and for asking whether disappointing results point to a flawed program rather than a flawed population. The conversation touches on the work of Jean Piaget and Maria Montessori as early models of culturally responsive practice and closes with a bigger idea: that done well, evaluation isn't just a technical exercise — it's a path toward social justice and mutual learning for evaluators and communities alike. Additional notes from this episode can be found here . The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

What if the most tedious parts of your research workflow could shrink from months to minutes? In this episode, hosts Stacey Merola and William Pate sit down with Ron Wilson, Assistant Graduate Program Director for the GIS program at the University of Maryland Baltimore County, to explore how AI is reshaping the way researchers and educators work. Ron shares how AI went from helping him stress-test research ideas to slashing coding time, generating synthetic datasets, and building classroom visualizations in 20 minutes flat. But it's not all hype — Ron also gets candid about responsible AI use, the importance of prompt engineering, and why treating AI as a tool rather than a revolution is the key to getting real results. Whether you're a data professional, educator, or just AI-curious, this episode is packed with practical insight. Links to additional information For a summary of this episode, check out the notes here. Learn more about the UMBC GIS graduate program here , including their GIS Master of Professional Studies which is a two-year (18 credits) program designed for early-to-mid-career professionals seeking a more foundational knowledge of GI that goes beyond learning the basics of technology or application of analytical tools. For more information about this topic, check out the UMBC Mic’d Up Podcast where they recently released an episode on GIS and AI . The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

What does it mean to be “DOGEd”? In this powerful episode of The Method to Our Madness Podcast, hosts Stacey and Bill sit down with sociologist Tara Fannon, Ph.D. to unpack the human impact behind the 2025 federal funding cuts—and the often-overlooked fallout for government contractors. Drawing from her GovCon Stories Project, Tara shares insights from a mixed-methods study of 92 surveyed contractors and in-depth interviews with 20 individuals affected by sudden layoffs. What emerges is a deeply personal look at job loss in a sector once seen as stable: stories of disrupted identities, financial strain, lost healthcare, and the emotional toll of feeling invisible and disposable. This episode goes beyond the headlines to explore: The psychological impact of being laid off from mission-driven work How poor communication and lack of support compounded the crisis Why long-held assumptions about stability in government contracting no longer hold true How contractors are rebuilding—through entrepreneurship, upskilling, and new career paths Despite the challenges, this conversation also highlights resilience. Many impacted professionals remain committed to public-service values while reimagining what equitable and sustainable work could look like in the future. If you care about workforce equity, the future of government contracting, or the human stories behind economic disruption, this episode is a must-listen. A downloadable summary of the episode can be found here: Episode 14 notes If you are interested in finding out more about services provided by Mind the Movement, their website can be found here: Mind the Movement Podcast music written and produced by Stacey Merola. Podcast artwork created by Zoe Targino.

In Episode 13 of The Method to Our Madness Podcast, we sit down with evaluation management expert Kirk Knestis to unpack the real-world challenges of managing evaluation firms and projects—and how to handle them like a pro. From avoiding scope creep to setting up airtight quality assurance processes, Kirk shares practical strategies that can make or break an evaluation project. He dives into why separating study protocols from contracts is essential, how to establish clear expectations with clients, and what it really takes to run evaluation work as a sustainable business. Through two cautionary “horror stories,” Kirk reveals common pitfalls—from misaligned reporting expectations to misunderstandings of statistical results—that can be problematic for even the most well-designed evaluations. Whether you’re leading projects, managing teams, or transitioning into independent consulting, this episode offers essential lessons on protecting your work, maintaining flexibility, and navigating client relationships with confidence. A longer summary of the episode can be found at Episode 13 Notes. Podcast music written and produced by Stacey Merola. Podcast artwork designed and created by Zoe Targino.

What does it take to prepare the next generation of evaluators to meet the challenges of a changing world? This episode dives into the shifting demands on the field, from federal funding uncertainty to the growing influence of AI tools, and the skills future evaluators will need to rise to the challenges. Our guests from the University of Minnesota discuss how future evaluators should be trained to think systemically, challenge assumptions, and act as collaborative problem-solvers, as well as possibly be prepared for an employment model based on project work, rather than extended tenure at one organization. Whether you're in education, public policy, or nonprofit work, this conversation offers timely insight into how evaluation is being redefined for the future. Our guests this week from the University of Minnesota are: John Hitchcock, Ph.D. , Professor of Evaluation Studies and Marilyn Sime Professor for Educational Excellence, hitch141@umn.edu Jodi Benenson, Ph.D. , Associate professor in the Department of Organizational Leadership, Policy, and Development in the College of Education and Human Development, benenson@umn.edu Stuart Yeh, Ph.D. , Professor and Coordinator of Graduate Programs in Evaluation Studies, yehxx008@umn.edu We cover a lot of topics in this episode, so if desired, a written summary can be found here: Episode 12 summary Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

Logic models don’t have to be complicated or static. In this conversation with guest Deborah Levy , Stacey and Bill discuss why simplicity matters, how logic models should evolve over time, and what happens when programs face unexpected challenges like shifting audiences or external disruptions. The episode also offers practical tips for engaging stakeholders, collecting data, and using logic models as living tools throughout an evaluation. Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

Fidelity of implementation is often talked about but rarely understood. In Episode 10 of The Method to Our Madness, Allan Porowski, Principal Associate at Abt Global, brings clarity to this critical concept in program evaluation. Drawing on years of experience and large‑scale evaluation work, Allan explains why fidelity metrics must be tailored to each program, how planned‑offered‑received frameworks reveal meaningful variation, and why imperfect implementation can still drive valuable insight. This episode offers practical wisdom for evaluators seeking to interpret results with context, nuance, and purpose. Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

In this episode of The Method to Our Madness, Stacey and Bill unpack the different types of statistics commonly used in program evaluation—and why “simple” doesn’t mean unsophisticated. They explore how evaluators balance accessibility with rigor, discussing when descriptive statistics, t-tests, and ANOVA are the right tools, and when more advanced methods like hierarchical linear modeling or propensity score matching make sense. Along the way, they dig into the hidden complexity behind clean-looking results, the limits of common data visualization tools, and why effect sizes often matter more than p-values. Drawing on real research experiences, disciplinary perspectives from sociology and psychology, and lessons learned from both practice and academia, this conversation highlights what it really takes to produce statistical findings that are both credible and useful. The episode wraps up with a reflection on communicating results clearly—and an open invitation for listeners to suggest future topics they’d like to explore in greater depth. Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

What’s the difference between formative, process, outcome, and impact evaluation? When should you actually use each one? In this episode, Stacey and Bill break down the why and when of evaluation, drawing on real-world examples from education, training programs, and policy work. They talk implementation fidelity, stakeholder engagement, logic models, and why not every evaluation is about proving causality. The takeaway: good evaluation is about learning, improvement, and usefulness—not just measurement. Music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

In this episode of The Method to Our Madness Podcast, we sit down with Rebecca Pitkow , a business development consultant with over 20 years of experience, to break down what actually works in proposal writing and procurement strategy. We dive into how to identify the right opportunities early, avoid aspirational bidding, and use “gates” as decision points to save time, money, and energy. Rebecca explains how proactive forecasting, realistic capability assessments, and early disqualifications can dramatically improve win rates—especially in today’s tight fiscal environment. The conversation also covers proposal color reviews (Pink Team through White Glove), building strong win themes and distinguishers, and conducting gap analyses to determine whether you should be a prime or a partner on a bid. Rebecca shares actionable advice on scaling proposal processes, developing reusable templates, and using post-mortem reviews to continuously improve. We wrap up with practical guidance on cold calling, relationship-building, and managing micro-procurements efficiently—plus how small businesses can bring discipline and structure to their proposal efforts. Rebecca closes by sharing how her consulting services help organizations build smarter, more effective proposal processes. If you want help with your business development methodology, can reach out to Rebecca directly at: rebecca.f.pitkow@gmail.com . Podcast music by Stacey Merola. Podcast artwork by Zoe Targino.

Ever gone to a conference for the free swag and accidentally left with a career‑changing conversation? From data‑themed outfits to awkwardly empty presentation rooms, Stacey and Bill swap stories about how to get the most out of your conference experience. Drawing on years of attending, presenting at, and navigating professional and academic conferences across evaluation, psychology, sociology, and research spaces, Stacey and Bill share honest reflections about what worked, what didn’t, and what they wish they’d known sooner. In this episode, they cover: Why people go to conferences in the first place How networking leads to jobs and collaborations Big conferences vs. small ones, and which might be better for you Simple ways to be more approachable (and memorable) The highs and lows of presenting, from packed rooms to empty ones How location, planning, and conference structure shape the experience Episode takeaways: Conferences are most valuable when you’re intentional about why you’re attending. Smaller or regional conferences often lead to more meaningful conversations. Being prepared—and approachable—can open unexpected doors. Not every presentation will be a hit, and that’s okay. Serendipity plays a bigger role than most people expect. Links to organizations and conferences mentioned in this episode: American Evaluation Association (AEA): https://www.eval.org/ American Psychological Association (APA): https://convention.apa.org/ Eastern Evaluation Research Society (EERS): https://www.eers.org/ Society for Industrial and Organizational Psychology (SIOP): https://www.siop.org/events/the-annual-conference/ Society for Research on Educational Effectiveness (SREE): https://www.sree.org/2026-conference Mixed Methods International Research Association (MMIRA): https://mmira.wildapricot.org/australia2026 Podcast music written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino.

Episode 05: Mixed methods with special guest John Hitchcock Have you ever wanted to know about using mixed methods in your research? Check out our latest podcast on mixed methods and the paradigm wars in program evaluation for a discussion about what mixed methods is and is not with special guest John Hitchcock, Marilyn Sime Professor for Educational Excellence at the University of Minnesota. John Hitchcock, a colleague of both Stacey and Bill, has published extensively on the topic mixed methods research. He explains what mixed methods is and is not in the world of evaluation research, shares examples from his work and collaborations with other leaders in the field and answers our questions about the topic. Below are resources referenced by John and additional information on colleagues and concepts discussed in this episode. Burke Johnson Google Scholar page Mixed methods research: A research paradigm whose time has come (2004) Article co-written with Anthony J. Onwuegbuzie Anthony (Tony) J. Onwuegbuzie University of Cambridge faculty bio Paradigm shift Popularized by Thomas Kuhn in The Structure of Scientific Revolutions (1962; Wikipedia article ) John W. Creswell and Vicki L. Plano Clarn Designing and Conducting Mixed Methods Research (Sage College Publishing) Michael Fetters Bio (Michigan Mixed Methods Program) The Mixed Methods Research Workbook (Sage Publications) Bonnie MacDougall Sri Lankan village life (Cornell Chronicle, July 17, 2018) Visit us on show website: https://themethodtoourmadness.podbean.com/ Instagram: https://www.instagram.com/methodtoourmadnesspodcast/ Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

In this podcast episode we discuss the principles that guide the protection of human subjects in medical and behavioral research by reviewing one of the seminal pieces on the subject, The Belmont Report. We also explore human subjects' protections in the context of current technological innovations. Notes related to the episode can be found here . Music written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino.

In this episode, we go to Guatemala to chat with UX expert Anna Sandoval, Ph.D. about UX and qualitative research. UX research is the process of understanding user perceptions, needs, and behaviors, in order to design products that are accessible and usable. Gaining this understanding involves collecting data through qualitative methods like interviews and observations. After listening to the episode, if you are interested in learning more, you can contact Anna on LinkedIn at Anna Sandoval Giron, PhD | LinkedIn or at Anna.b.sandoval@gmail.com . Podcast music written and produced by Stacey Merola. Podcast artwork by Zoe Targino.

Join us as we explore the roots of evaluation and the Evidence Act in this episode. Notes and sources are available here: https://drive.google.com/file/d/1pLF4kYfyg25Ew3nQ4yD6tP6-AVnRPY3G/view?usp=drive_link . Music written and produced by Stacey Merola. Artwork by Zoe Targino.