Neural Nets for NLP 2021 - Document-Level Models

Learn about document-level natural language processing tasks and models in this lecture from CMU's Neural Networks for NLP course. Explore techniques for handling long documents, including sentence-level and document-level tasks, entity coreference resolution, and discourse parsing. Examine approaches like infinitely passing state, separate encoding for document context, and self-attention across sentences. Dive into specific models such as Transformer-XL, Adaptive Span Transformers, and Reformer. Understand the components and challenges of coreference resolution, including mention detection and different modeling approaches. Discover how neural network models can be applied to coreference tasks and discourse parsing. Gain insights into evaluating document-level models and leveraging discourse structure in neural approaches.

Syllabus

Some NLP Tasks we've Handled
Some Connections to Tasks over Documents
Document Level Language Modeling
Remember: Modeling using Recurrent Networks
Simple: Infinitely Pass State
Separate Encoding for Coarse- grained Document Context
Self-attention/Transformers Across Sentences
Transformer-XL: Truncated BPTT+Transformer
Adaptive Span Transformers
Reformer: Efficient Adaptively Sparse Attention
How to Evaluate Document- level Models?
Document Problems: Entity Coreference
Mention(Noun Phrase) Detection
Components of a Coreference Model
Coreference Models:Instances
Mention Pair Models
Entity Models: Entity-Mention Models
Advantages of Neural Network Models for Coreference
End-to-End Neural Coreference (Span Model)
End-to-End Neural Coreference (Coreference Model)
Using Coreference in Neural Models
Discourse Parsing w/ Attention- based Hierarchical Neural Networks
Uses of Discourse Structure in Neural Models