class: center middle main-title section-title-3 # In-person<br>session 4 .class-info[ **February 5, 2026** .light[PMAP 8521: Program evaluation<br> Andrew Young School of Policy Studies ] ] --- name: outline class: title title-inv-8 # Plan for today .box-6.medium[More regression things] -- .box-1.medium[Logic models and evaluation] -- .box-2.medium[Measuring outcomes] -- .box-5.medium[DAGs] --- layout: false name: exam class: center middle section-title section-title-4 animated fadeIn # Exam 1 details --- layout: false name: exam class: center middle section-title section-title-4 animated fadeIn # Final project details --- layout: false name: regression-faqs class: center middle section-title section-title-6 animated fadeIn # More regression things --- layout: true class: middle --- .box-6.medium[Interactive null worlds] --- .box-6.medium[Statistical vs.<br>substantive significance] --- layout: false name: logic-models class: center middle section-title section-title-1 animated fadeIn # Logic models<br>and evaluation --- class: title title-1 # The paradox of evaluation -- .box-1.medium[Evaluation is good, but expensive] .box-inv-1.sp-after.SMALL["Evaluation thinking"] -- .box-1.medium[Too much evaluation is bad] .box-inv-1.sp-after-half.SMALL[Taming programs] --- layout: false class: middle .box-1.large[Logic model] .box-inv-1[Let's make another logic model!] --- class: middle .pull-left[ <figure> <img src="img/04-class/DPIL-DP-RockstarWeb2.png" alt="Dolly Parton's Imagination Library" title="Dolly Parton's Imagination Library" width="100%"> </figure> ] .pull-right[ .box-1.medium[Dolly Parton's<br>Imagination Library] .box-inv-1.tiny[[Main site](https://imaginationlibrary.com/)] .box-inv-1.tiny[[Video intro](https://www.youtube.com/watch?v=tscb01jfukc)] ] ??? - <https://imaginationlibrary.com/> - <https://imaginationlibrary.com/international-research/> --- layout: false name: outcomes class: center middle section-title section-title-2 animated fadeIn # Measuring<br>outcomes --- layout: true class: title title-2 --- # Outcomes and programs .box-2.medium[Outcome variable] .box-inv-2.sp-after-half.SMALL[Thing you're measuring] -- .box-2.medium[Outcome change] .box-inv-2.sp-after-half.SMALL[∆ in thing you're measuring over time] -- .box-2.medium[Program effect] .box-inv-2.SMALL[∆ in thing you're measuring over time *because of* the program] --- # Outcomes and programs <figure> <img src="img/04/program-effect.png" alt="Outcomes and program effect" title="Outcomes and program effect" width="100%"> </figure> --- layout: false class: middle .box-2.large[Abstraction] .box-inv-2[Let's abstract an outcome!] .box-inv-2.tiny[[Ladder of abstraction for witches](https://evalsp26.classes.andrewheiss.com/slides/04-slides.html#20)] --- layout: false name: dags class: center middle section-title section-title-5 animated fadeIn # DAGs --- layout: true class: middle --- .box-5.large[Causal thinking is necessary—<br>even for descriptive work!] --- .center[ <figure> <img src="img/04-class/supercentenarians.png" alt="Supercentenarians" title="Supercentenarians" width="46%"> </figure> ] ??? Necessity of causal thinking: Mention the McElreath tweet on birth certificate introduction and death ages: <https://twitter.com/rlmcelreath/status/1427564280744976384> <https://www.biorxiv.org/content/10.1101/704080v2> --- layout: false .box-5.less-medium["Every time I get a haircut, I become more mature!"] <figure> <img src="img/04-class/ben-hair.jpg" alt="Benjamin haircut" title="Benjamin haircut" width="100%"> </figure> --- .box-5.less-medium["Every time I get a haircut, I become more mature!"] <img src="04-class_files/figure-html/unnamed-chunk-1-1.png" alt="" width="70%" style="display: block; margin: auto;" /> .center[$$E[\text{Maturity} \mid \operatorname{do} (\text{Get haircut})]$$] --- .box-5.less-medium[Getting older opens a backdoor path] <img src="04-class_files/figure-html/unnamed-chunk-2-1.png" alt="" width="70%" style="display: block; margin: auto;" /> --- .box-5.medium[Ice cream causes crime] -- .center[ <figure> <img src="img/05-class/eat-ice-cream-do-crime.jpg" alt="Eat ice cream, do crime" title="Eat ice cream, do crime" width="100%"> </figure> ] -- .box-inv-5[Summer weather opens a backdoor path] ??? - <https://twitter.com/EpiEllie/status/1080680304581451778> - <https://www.redbubble.com/i/sticker/Eat-Ice-Cream-Do-Crime-by-epiellie/150075475.EJUG5> --- class: middle .box-5.medium[But what does that mean,<br>"opening a backdoor path"?] .box-5.medium[How does statistical association<br>get passed through paths?] --- .box-5.less-medium.sp-after[How do I know which of these is which?] .center[ <figure> <img src="img/05-class/dag-associations.png" alt="DAG associations" title="DAG associations" width="100%"> </figure> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/confounding-dag-alone-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/mediating-dag-alone-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] --- class: middle .pull-left[ <figure> <img src="img/04-class/slider-switch-plain-80.jpg" alt="Switch and slider" title="Switch and slider" width="100%"> </figure> ] .pull-right[ <img src="04-class_files/figure-html/colliding-dag-alone-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] --- .center[ <video controls loop> <source src="img/05-class/video-confounding-unblocked.mp4" type="video/mp4"> </video> ] --- .center[ <video controls loop> <source src="img/05-class/video-confounding-blocked.mp4" type="video/mp4"> </video> ] --- .center[ <video controls loop> <source src="img/05-class/video-mediating.mp4" type="video/mp4"> </video> ] --- layout: true class: middle --- .box-5.medium[d-separation] .box-inv-5[Except for the one arrow between X and Y,<br>no statistical association can flow between X and Y] .box-inv-5[This is **identification**—<br>all alternative stories are ruled out<br>and the relationship is isolated] --- .box-5.medium[How do we know if we have the right DAG?] -- .box-5.medium[Right nodes? Enough nodes?<br>Too many nodes?] -- .box-5.medium.sp-after[Right arrows?] -- .box-inv-5.large[lol] --- .center.tiny[Nick Huntington‐Klein et al., “The Influence of Hidden Researcher Decisions in Applied Microeconomics,”<br>*Economic Inquiry* 59, no. 3 (2021): 944–60, <https://doi.org/10.1111/ecin.12992>.] .center[ <figure> <img src="img/04-class/many-analysts.png" alt="Many analysts" title="Many analysts" width="55%"> </figure> ] --- layout: true class: middle --- .box-5.large[What's the difference between<br>logic models and DAGs?] .box-inv-5[Can't I just remake my logic model in Dagitty and be done?] --- layout: false class: title title-5 # DAGs vs. Logic models .box-5.large[DAGs are a *statistical* tool] .box-inv-5[Describe a data-generating process<br>and isolate/identify relationships] -- .box-5.large.sp-before[Logic models are a *managerial* tool] .box-inv-5[Oversee the inner workings of a program and its theory] --- layout: true class: middle --- .box-5.large[How exactly do colliders<br>mess up your results?] .box-5.medium[It looks like you can<br>still get the effect of X on Y] --- <img src="04-class_files/figure-html/nba-dag-1.png" alt="" width="65%" style="display: block; margin: auto;" /> --- .center[ <figure> <img src="img/04-class/facebook.png" alt="Facebook collider" title="Facebook collider" width="55%"> </figure> ] ??? <https://www.businessinsider.com/facebook-sent-incomplete-misinformation-data-flawed-researchers-2021-9> <https://www.nytimes.com/live/2020/2020-election-misinformation-distortions#facebook-sent-flawed-data-to-misinformation-researchers> --- layout: false class: title title-5 # Does niceness improve appearance? .pull-left[ <img src="04-class_files/figure-html/unnamed-chunk-3-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="04-class_files/figure-html/unnamed-chunk-4-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] --- class: title title-5 # Collider distorts the true effect! .pull-left[ <img src="04-class_files/figure-html/unnamed-chunk-5-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="04-class_files/figure-html/unnamed-chunk-6-1.png" alt="" width="100%" style="display: block; margin: auto;" /> ] --- .box-5.large[Effect of race on police use of force<br>using administrative data] --- .box-5.medium[Effect of race on police use of force<br>using administrative data] .pull-left[ <figure> <img src="img/05-class/klm-dag.png" alt="Use of force" title="Use of force" width="100%"> </figure> ] .pull-right[ <figure> <img src="img/05-class/klm.png" alt="Use of force" title="Use of force" width="100%"> </figure> ] --- layout: false class: middle .box-5.large[Let's make a DAG!]