Department of Computer Science Colloquium Series
NLP in the Time of Generative Models: Centering the Role of People with Sachin KumarAbstract
Every language, like its speakers, is immensely diverse. Deep learning-based natural language processing models, even when trained on such heterogeneous data, are generally monolithic, encoding only the majority signal and averaging over all other variations. As a result, they consistently fail to support language use outside the “standard” and present challenges to the models' equitable access. In this talk, I discuss this issue in the context of generative models for text and describe how these shortcomings can be addressed by developing new adaptable and controllable training and inference algorithms.
In the first part of the talk, I describe training algorithms for text generation that separate token representation learning from model learning resulting in improved lexical diversity in the generated text and easy adaptability to generate related language varieties. I then introduce inference algorithms from pre-trained language models to control for stylistic and structural variations. I frame text generation as constrained optimization with gradient-based methods to generate text non-autoregressively, updating the entire output sequence iteratively. I conclude by introducing my recent work on diffusion-based text generation models that have controllability baked in.
Sachin Kumar is a Ph.D. Candidate at the Language Technologies Institute at Carnegie Mellon University, advised by Prof. Yulia Tsvetkov and a visiting researcher at the Paul G Allen School of Computer Science at the University of Washington, Seattle. His research broadly revolves around Machine Learning and Natural Language Processing (NLP) with a particular focus on algorithms for user-adaptable and controllable language generation as well as fair, robust, and grounded language understanding. His work is supported by a Google Ph.D. Fellowship. Prior to starting his Ph.D., Sachin received his undergraduate degree (BTech summa cum laude) in Computer Science from the Indian Institute of Technology Kharagpur.