CMU Advanced NLP: Experimental Design

Explore experimental design in advanced natural language processing through this comprehensive lecture. Learn about the scientific method, formulating research questions, conducting literature surveys, and developing hypotheses. Discover techniques for running experiments, including data collection, annotation guidelines, and statistical significance testing. Gain insights into identifying promising research directions, from curiosity-driven approaches to application-focused inquiries. Understand the importance of proper experimental setup and evaluation metrics in NLP research. Access valuable resources for finding relevant papers and datasets to support your investigations.

Syllabus

Intro
Identify good research directions
Examples of applications
Curiosity driven research
Bottomup discovery
Research survey methods
Where to find papers
ACL Anthology
Google Scholar
Resources
Research Question
Hypothesis
Application Driven Questions
Testing with Experiments
Obtaining Test Data
Finding Datasets
Annotation
Statistical significance testing
Sample data
Annotation guidelines
Annotation methods
Measure metrics