Stanford CS 329X | Human-Centered NLP

Schedule

Note: tentative schedule is subject to change.

Week	Date	Theme	Course Material
1	Apr 3 Monday	What is Human-Centered NLP [slides] [scribe template]
1	Apr 5 Wednesday	The Ultimate Crash: NLP Tasks & Applications [slides] Basics of language models, pretraining, fine-tuning and prompting	Optional Readings: Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein et al. "On the opportunities and risks of foundation models." arXiv preprint arXiv:2108.07258 (2021). Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. "Chain of thought prompting elicits reasoning in large language models." arXiv preprint arXiv:2201.11903 (2022).
2	Apr 10 Monday	Design Thinking [slides] [scribe] Overview of user-centered design, value-centered design and other design thinking	Optional Readings: Pea, Roy D. "User centered system design: new perspectives on human-computer interaction." Journal educational computing research 3, no. 1 (1987): 129-134. Friedman, Batya, David G. Hendry, and Alan Borning. "A survey of value sensitive design methods." Foundations and Trends® in Human–Computer Interaction 11, no. 2 (2017): 63-125. Birhane, Abeba, William Isaac, Vinodkumar Prabhakaran, Mark Díaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. "Power to the People? Opportunities and Challenges for Participatory AI."Equity and Access in Algorithms, Mechanisms, and Optimization (2022): 1-8.
2	Apr 12 Wednesday	Human in the Loop [slides] [scribe] [project signup] Homework is out on Canvas. Human or machine in the loop, where and who are the humans? Interactive NLP systems	Optional Readings: Wang, Zijie J., Dongjin Choi, Shenyu Xu, and Diyi Yang. "Putting humans in the natural language processing loop: A survey." arXiv preprint arXiv:2103.04044 (2021). Amershi, Saleema, Maya Cakmak, William Bradley Knox, and Todd Kulesza. "Power to the people: The role of humans in interactive machine learning." Ai Magazine 35, no. 4 (2014): 105-120.
3	Apr 17 Monday	Learning from Human Feedback [slides] [scribe] Different forms of feedback, and approaches to incorporate human feedback; RLHF	Optional Readings: Stiennon, Nisan, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021. Ouyang, Long, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang et al. "Training language models to follow instructions with human feedback." arXiv preprint arXiv:2203.02155 (2022). ChatGPT: Optimizing Language Models for Dialogue by OpenAI in 2022
3	Apr 19 Wednesday	📺 Guest Lecture: Interactive Visualization [slides] [scribe] Sherry Wu
4	Apr 24 Monday	Human-Centered Evaluation [slides] Evaluation methods and metrics; best practices around quantitative and qualitative evaluations 🔥 Fireside Chat w/ Mina Lee	Optional Readings: Schuff, Hendrik, Lindsey Vanderlyn, Heike Adel, and Ngoc Thang Vu. "How to do human evaluation: A brief introduction to user studies in NLP." Natural Language Engineering (2023): 1-24. Lee, Mina, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin et al. "Evaluating Human-Language Model Interaction." arXiv preprint arXiv:2212.09746 (2022). Tulio Ribeiro, Marco, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. "Beyond Accuracy: Behavioral Testing of NLP models with CheckList." arXiv e-prints (2020): arXiv-2005.
4	Apr 26 Wednesday	Data Collection and Curation [slides] [scribe] Annotation task design; annotation paradigms; dataset artifacts and difficulty 🔥 Fireside Chat w/ Mitchell Gordon	Optional Readings: Gururangan, Suchin, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. "Annotation artifacts in natural language inference data." arXiv preprint arXiv:1803.02324 (2018). Gordon, Mitchell L., Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. "Jury learning: Integrating dissenting voices into machine learning models." In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19. 2022. Röttger, Paul, Bertie Vidgen, Dirk Hovy, and Janet B. Pierrehumbert. "Two contrasting data annotation paradigms for subjective nlp tasks." arXiv preprint arXiv:2112.07475 (2021).
5	May 1 Monday	Beyond Benchmarking [slides] [scribe] The role of benchmarking; what benchmarks can and can't do; rethinking benchmark 🔥 Fireside Chat w/ Alex Tamkin	Optional Readings: GKiela, Douwe, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen et al. "Dynabench: Rethinking benchmarking in NLP." arXiv preprint arXiv:2104.14337 (2021). Sambasivan, Nithya, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M. Aroyo. "“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI." In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021.
5	May 3 Wednesday	Course Project Showcase 🤩
6	May 8 Monday	Guest Lecture: Ethics and Social Responsibility [slides] Rishi Bommasani
6	May 10 Wednesday	Values and Culture [slides] Values, norms and culture encoded in (gen)AI. 🔥 Fireside Chat w/ Pratyusha Ria Kalluri	Optional Readings: Birhane, Abeba, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. "The values encoded in machine learning research." In 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 173-184. 2022. Birhane, Abeba, William Isaac, Vinodkumar Prabhakaran, Mark Díaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. "Power to the People? Opportunities and Challenges for Participatory AI." Equity and Access in Algorithms, Mechanisms, and Optimization (2022): 1-8. Hershcovich, Daniel, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello et al. "Challenges and Strategies in Cross-Cultural NLP." ACL 2022.
7	May 15 Monday	Guest Lecture + Fireside Chat 🔥 Privacy and Security [slides] Eric Wallace	Optional Readings: Carlini, Nicholas, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts et al. "Extracting Training Data from Large Language Models." In USENIX Security Symposium, vol. 6. 2021. Ganguli, Deep, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).
7	May 17 Wednesday	Guest Lecture: Architecting Proxies of Our Lives in Simulations. Joon Sung Park	Optional Readings: Park, Joon Sung, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. "Social Simulacra: Creating Populated Prototypes for Social Computing Systems." In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1-18. 2022. Horton, John J. "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?." arXiv preprint arXiv:2301.07543 (2023).
8	May 22 Monday	Trust and Social Impact [slides] Ways to understand and quantify trust; applications of NLP for social good; societal issues; inclusion 🔥 Fireside Chat w/ Elisa Kreiss	Optional Readings: Jacovi, Alon, Ana Marasović, Tim Miller, and Yoav Goldberg. "Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI." In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 624-635. 2021. Yin, Kayo, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. "Including signed languages in natural language processing." arXiv preprint arXiv:2105.05222 (2021). Kreiss, Elisa, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, and Christopher Potts. "Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics." arXiv preprint arXiv:2205.10646 (2022).
8	May 24 Wednesday	Human AI Collaboration [slides] Human-AI interaction; human-AI teaming and collaboration; augmenting human capabilities	Optional Readings: Bansal, Gagan, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. "Does the whole exceed its parts? the effect of ai explanations on complementary team performance." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-16. 2021. Lee, Mina, Percy Liang, and Qian Yang. "Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities." In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19. 2022. Gero, Katy Ilonka, Vivian Liu, and Lydia Chilton. "Sparks: Inspiration for science writing using language models." In Designing Interactive Systems Conference, pp. 1002-1019. 2022.
9	May 29 Monday	Memorial Day - No Class
9	May 31 Wednesday	📺 Guest Lecture: AI Governance Irene Solaiman	Optional Readings: Solaiman, Irene. "The gradient of generative AI release: Methods and considerations." arXiv preprint arXiv:2302.04844 (2023). The Gradient Podcast, Irene Solaiman: AI Policy and Social Impact, April 2023.
10	June 5 Monday	🔥 Fireside Chat: AI, People, and Society Eric Horvitz and Peter Norvig	Optional Readings: Lecture on AI and Human Values Horvitz, Eric. "One Hundred Year Study on Artificial Intelligence: Reflections and Framing."
10	June 7 Wednesday	In-Class Final Project Presentation

CS 329X: Human-Centered NLP

Stanford / Spring 2023

Instructors

Welcome!

Schedule

Overview

Course Info

Office Hours

Prerequisites

Academic Accommodations

Well-Being, Stress Management, & Mental Health