Meeting recap
The July Research Forum was intentionally exploratory. The point was not to present a finished project, but to expose the Open Inference Project early enough that colleagues could challenge the premise, the measurement strategy, and the path from a dermatology corpus to a broader research program.
A special thank-you to colleagues whose questions, critiques, and perspectives helped shape the conversation around analytic search space and the Open Inference Project.
David M. Miller Lisa Zaba Vishal Patel Isaac Brownell
The Open Inference Project
Making the analytic context of evidence visible
The conjecture
Scientific results do not arrive without context. Their meaning depends on the questions asked, the populations and outcomes examined, the models that were fit, and the alternative analytic paths that were available.
This Research Forum presentation began with a simple conjecture: the analytic context of evidence is systematically underappreciated. Published results are highly visible, while much of the analytic process that produced them can be difficult for readers to reconstruct.
Central Idea
The problem is not analytic complexity. The problem is analytic complexity that readers cannot see.
Where analytic search space comes from
Multiple outcomes
Primary, secondary, exploratory, and time-dependent outcomes create multiple reasonable questions within the same study.
Multiple models
Covariate sets, functional forms, model specifications, and variable definitions can each change the analysis.
Multiple subgroups
Age, disease severity, biomarkers, and other patient characteristics create additional analytic pathways.
Multiple decisions
Inclusion criteria, missing-data handling, transformations, and other defensible choices expand the space of possible analyses.
Studying the analytic structure of a literature
Rather than treating analytic flexibility and multiplicity as purely theoretical concerns, the project asked an empirical question: how large is the analytic search space in contemporary dermatology research?
An initial manual review suggested a substantial analytic search space. The work was then expanded into a structured, reproducible pipeline designed to characterize statistical reporting patterns across a much larger corpus of articles from the Journal of the American Academy of Dermatology.
Early Signal
Across the JAAD corpus, statistical testing is highly visible, while multiplicity adjustment, preregistration, and alternative inferential frameworks appear far less often.
What the review suggests
- Researchers often have many reasonable ways to analyze the same data.
- P-values are extremely common in the published dermatology literature.
- Adjustments for multiple hypothesis testing are comparatively uncommon.
- Preregistration is rare outside clinical trials.
- Bayesian approaches are rarely used.
- Analytic flexibility frequently remains implicit rather than directly visible to the reader.
What can we do about it?
Investigator level
Improve how individual studies are designed, analyzed, interpreted, and shared. Define questions clearly, distinguish confirmation from exploration, prespecify consequential choices when feasible, and communicate uncertainty.
Systems level
Move beyond transparency within one paper and make the analytic structure of an entire evidence base easier to discover, examine, compare, and interpret.
A Two-Level Response
Transparent individual studies are necessary, but they do not by themselves make the structure of a literature visible.
The Open Inference Project
The Open Inference Project is a research program focused on the structure behind published evidence. It shifts attention from an isolated reported result to the larger analytic context that produced it.
The goal is to make features such as analytic flexibility, multiplicity, preregistration, robustness, and interpretability more visible across the scientific literature—and ultimately to help readers understand not only what a study found, but the analytic context in which that result should be interpreted.
The Open Inference Project
Better inference requires making the analytic context of evidence visible.
Why bring this to the Research Forum?
The presentation was intentionally framed as work in progress. The empirical analysis, measurement strategy, and broader Open Inference framework were presented while they were still open to criticism and revision.
That reflects the purpose of the SoCO Research Forum itself: bringing ideas forward early enough that colleagues can challenge assumptions, expose weaknesses, and improve the work before the story becomes fixed.
Who joined us?
The attendance record is preserved as part of the meeting recap. The roster below is limited to names and institutional affiliations.
Attendance roster View names and affiliations
This roster records meeting attendance and institutional affiliation only.
| Name | Affiliation |
|---|---|
| Adewole S. Adamson | The University of Texas at Austin |
| Adewunmi O. Adelaja | Beth Israel Lahey |
| Ajay N. Sharma | Mass General Brigham |
| Aleigha R. Lawless | Mass General Brigham |
| Bailey Claire Smith | Mass General Brigham |
| Ben Gratz | University of Washington |
| Charlie Li | Washington University in St. Louis |
| Dan Hippe | University of Washington |
| David J Savage | University of New Mexico Comprehensive Cancer Center |
| David M. Miller | Mass General Brigham |
| Emily Godwin Summers | University of North Carolina |
| Frances Collichio | University of North Carolina |
| Haroutyun Joulfayan | Washington University School of Medicine in St. Louis |
| Howard L. Kaufman | Mass Eye and Ear |
| Isaac Brownell | National Institutes of Health |
| Itai M. Pashtan | Dana-Farber Cancer Institute |
| Jack Kollings | University of Washington |
| Jessica L. Fewkes | Mass Eye and Ear |
| Juliane Andrade Czapla | Mass General Brigham |
| Justine V. Cohen | Dana-Farber Cancer Institute |
| Kathryn Bollin | Scripps Health |
| Krista Lachance | University of Washington |
| Laura Ferris | University of North Carolina |
| Lisa Zaba | Stanford University |
| Mehran Behruj Yusuf | University of Alabama at Birmingham |
| Michael Povelaitis | University of North Carolina |
| Paul Nghiem | University of Washington |
| Peter Ch’en | University of Washington |
| Ray Cheever | University of North Carolina |
| Rhoda Alani | Mass General Brigham |
| Ross D. Merkin | Mass General Brigham |
| Sameer Gupta | Mass Eye and Ear |
| Shailender Bhatia | University of Washington |
| Song Park | University of Washington |
| Sonia Cohen | Mass General Brigham |
| Todd Franklin Pearson | Mass General Brigham |
| Vern Sondak | Moffitt Cancer Center |
| Vishal Patel | George Washington University |
Society of Cutaneous Oncology Research Forum. Research discussions presented through the Forum may include work in progress and should be interpreted in that context.