Generative models are a class of machine learning algorithms that operate over complex, high-dimensional objects such as images, sequences, and graphs. Recent advances have greatly improved the capabilities of generative models and have enabled new applications in computer-generated art, natural language processing, computational drug design, and other areas. This course explores the foundational probabilistic principles of generative models, their learning algorithms, and popular model families, which include variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, and diffusion models. The course also covers applications in domains such as computer vision, natural language processing, and biomedicine, and draws connections to the field of reinforcement learning.

First Lecture Information

The first lecture will be held on Monday 01/23 in Bloomberg 91 (streamed to Gates G13). The following Zoom link will also be available to connect to the lecture:

• Zoom link:   https://cornell.zoom.us/j/99817943690?pwd=TEQ3bHlXUnBhajd1ck50RFBTTmhXZz09

You may also find Zoom links to other lectures in Canvas under the "Zoom" tab.

Key Dates

Please note the following important dates for this semester:

• January 30: Assignment 1 is released
• February 13: Assignment 1 is due
• February 15: Assignment 2 is released
• February 22: Project proposal is due
• February 28: No class (February break)
• March 1: Assignment 2 is due.
• March 6: Assignment 3 is released
• March 20: Assignment 3 is due
• March 22 to April 26: Student presentations
• March 31: Project progress report is due
• April 3 & 5: No class (spring break)
• April 10: Assignment 4 is released
• April 24: Assignment 4 is due
• May 1 to May 9: Time off to work on projects
• May 15: Final project report is due

Please use this link to sign up for presentations: https://docs.google.com/spreadsheets/d/1RbjnwVAIIWjVVA9ZZgE6Q6nRqQHJlXEiD5RYE6GzFDk/edit?usp=sharing

Information and Instruction Format

Instructor: Volodymyr Kuleshov

Credits: 3

Times: Mon/Wed - 1:00-2:15pm Eastern Time.

Location: Bloomberg 91 (Tech Campus). Ithaca campus room is Gates G13.

The course will be offered in a hybrid in-person/remote format:

• The instructor will give the first several weeks of lectures in person from the Tech campus (Bloomberg 91). The lectures will be streamed to an Ithaca classroom (Gates G13) as well as on Zoom.
• Because of conference travel, several in-person lectures will be held remotely. Students will be notified ahead of time via Canvas.
• The last third of the course will consist of student presentations. Students can form teams across the two campuses and present over Zoom. We do not anticipate using the classrooms at that point, unless the students prefer to give their presentation from a classroom.

Teaching Staff and Office Hours

Volodymyr Kuleshov (Instructor). https://www.cs.cornell.edu/~kuleshov/

Office Hours: MW after class. 2:15-2:45pm. 

Prerequisites

• Basic knowledge about machine learning from at least one of: CS4780, CS4701, CS5785.
• Basic knowledge of probability and calculus: students will work with computational and mathematical models.
• Basic knowledge of deep neural networks (CNNs, RNNs; CS5787). Experience implementing deep neural networks is not required but will be helpful for the class project.
• Proficiency in a programming language (preferably Python).

Grading

Assignments

The course will feature three assignments/homeworks that will contain a mix of written and programming questions. Assignments will be released, submitted, and graded using Gradescope.

If you're not already in Gradescope, you can use this code to enroll: DJDR4E

The programming questions should be done in PyTorch using the provided starter code. You will be required to submit figures generated using your code as a separate PDF in Gradescope.

The written part of the homeworks should be written up clearly and succinctly; you may lose points if your answers are unclear or unnecessarily complicated. You are encouraged to use LaTeX to writeup your homeworks, but this is not a requirement. You can find a basic LaTeX template here. Assignments will be submitted on Gradescope.

You are encouraged to work in teams of two: make sure to put both of your names on the submission and submit as a team in Gradescope.

Course Project

The course project will give the students the chance to explore generative models in greater detail. Course projects will be done in groups of up to 3 students and can fall into one or more of the following categories:

• Application of machine learning to a practical problem or a dataset.
  
• Improvements to machine learning algorithms.
  
• Theoretical analysis of any aspect of machine learning models.
  

Pick a topic that's meaningful to you and that excites you. For example, if you do PhD research in biology, you can do a project related a dataset that you work with as part of your thesis research. If you're an Urban Tech student, you can work with an urban dataset that you find interesting. You are encouraged to find something on your own, but feel free to talk to the teaching team during office hours about project ideas.

Proposal

Your proposal should give the title of the project, the project category, the names of your team members, their NetID, and a 300-500 word description of what you plan to do. It should contain the following information.

• Motivation: What problem are you tackling? Is this an application or a theoretical result?
• Method: What machine learning techniques are you planning to apply or improve upon and how?
• Future work: What experiments are you planning to perform or what theorems do you want to prove?

The goal of the proposal is make sure you're on the right track. As long as you follow the above guidelines, you should do well.

Please submit the proposal via Gradescope and make sure to submit as a team.

Milestone 

The milestone submission should describe what you've accomplished so far, and briefly say what else you plan to do. The format should be the same as of the final project, with an approximate length of 3-5 pages (excluding references). The goal is to make sure that you are on track to finish the final project.

• Motivation: What problem are you tackling? Is this an application or a theoretical result?
• Method: What machine learning techniques are you planning to apply or improve upon and how?
• Preliminary experiments: Describe the experiments that you've run, the outcomes, and any error analysis that you've done. You should have tried at least one baseline.
• Future work: What else do you plan to do?

The goal of the milestone is make sure you're on the right track. As long as you follow the above guidelines, you should do well.

Please submit the milestone via Gradescope and make sure to submit as a team.

Final Writeup

The final writeup should describe all the work you did for your course project and summarize the main results. You can think of it as a technical report that presents your findings to a general machine learning audience.

The style and format of the writeup should be similar to that of a research paper. The maximum length is 8 pages, excluding references. 

There are no strict requirements on the structure of the final writeup, but one way to structuring it would be include the following sections, which are fairly standard for a research paper.

• Abstract: Summarize the problem, novel contributions, and results in one paragraph.
• Introduction: Provide motivation for the problem and expand upon the overview in the abstract.
• Background: Briefly summarize the background knowledge needed to understand the work.
• Method: Describe the methods that will be used or implemented in the paper.
• Theoretical analysis: If you are doing a theory project, describe your theoretical results here.
• Experimental analysis: Describe in detail your experiments.
• Discussion and Prior Work: Discuss the key takeaways from your experiments. Put your results in the context of previous work
• Conclusion. You may summarize the paper or talk about open problems and open directions.

Regardless of how the writeup is structured, please make sure to cover the following points.

• Motivation: What problem are you tackling? Why is it interesting? What type of project will this be (application, method, theory)?
• Method: What machine learning techniques are you planning to apply or improve upon and how? Make sure to describe them in detail and provide enough context for the reader to understand the methods at least at a high level. Provide any background that is necessary for that.
• Experiments: Describe the experiments that you've run, the outcomes, and any error analysis that you've done. Make sure that the setup is described in enough detail for someone else to reproduce your results. Also, if you have an experimental project, make sure to provide a detailed experimental analysis. Things you should consider including are: train/test performance, learning curves, model samples, error analyses, ablation analyses, etc. Most projects should also include baselines.
• Theory: If doing a theory project, state your results formally as theorems. Make sure that all the symbols are defined. Also, the best presentation of theoretical results tends to also explain the results in plain language and conveys the intuition behind them.
• Context: Explain how you build upon previous work and how your results compare to what has been done previously.

Writeups will be evaluated for their presentation clarity, the respect of the above guidelines, the significance of the project (does it explore a toy dataset or a real problem) and the technical quality of the work (the level of depth in the experimental or theoretical analyses, does the approach make sense technically, are the algorithms implemented reasonable and studied in enough detail, etc.).

Please submit the writeup via Gradescope and make sure to submit as a team.

Presentations

Students will be asked to deliver a presentation in the second half of the course. The expected length is 45-50 minutes, followed by a 10-15 min discussion. Presentations can be done individually or in groups of two and should cover 1-4 recent research papers.

Please use this link to sign up for presentations: https://docs.google.com/spreadsheets/d/1RbjnwVAIIWjVVA9ZZgE6Q6nRqQHJlXEiD5RYE6GzFDk/edit?usp=sharing

Topic Ideas: The best topics are ones that you find personally interesting. You can also choose from our list.

• The expected length of each presentation is 45-50 minutes. Presentations will be followed by 10-15 minutes of discussion. Students are asked to conclude their presentation with an initial set of discussion topics for the team and to contribute towards driving the discussion.
• The expected length of the reviews is up to 1 page each. These should be on a topic that will be presented and should be submitted before the presentation topic.
• There will be about ten presentation slots in the second half of the class. Students should email the instructor by [2/28] to reserve a presentation slot, choose a presentation topic, and choose one of the papers for reviews. Slots will be filled on a first come first served basis.
• Each presentation team needs to (1) send a presentation outline to the instructor at least two weeks before the talk and (2) send presentation slides at least two days before the talk.

Presentation Format

The ideal presentation will touch the following topics:

• Motivation for the problem being studied; why is it interesting?
• Context and previous work in this area;
• High-level summary of the novel ideas and contributions in the presented papers;
• Detailed explanation of the technical material;
• Summary of experimental or theoretical results;
• Discussion of the results;
• Conclusion and open-ended questions.

The presenting team should conclude the presentation with follow-up topics / questions to the audience to seed a discussion / q&a covering the presentation and future research directions. The team should drive the discussion, and the instructor will help as well.

Paper/Presentation Review

Additionally, each team should write a review of the papers presented by another team. The expected length is 1-2 pages, standard page formatting, 12pt font, single-spaced, submitted as a PDF in Gradescope. The review will count for 5% of the class grade. The presentation will count for 10% of the grade. Thus, both count for 15%.

We encourage you to submit the paper review as a team, preferably the with same team as for your presentation.

We ask you to adhere to the following timeline:

• Have a look at the presentation topics and select the one you would like to review. It has to be different from your presentation. Send the instructor and the TA the set of papers you will review by [Date TBD]. All the presentation topics will be finalized at least a week before then.
• In order to assure uniform coverage, each presentation has two review slots. In other words, at most two different teams can review the same set of papers. Review slots will be filled on a first come first serve basis.
• Please reserve your slot as soon as possible. Don't wait until the deadline.
• We ask you to submit your review the day before the presentation (by 11:59pm time zone of your choice). We want you to be already familiar with the topic of the presentation, and ask questions and/or give a different perspective on the presentation.
• We will be asking for mandatory attendance to the presentations. It is also mandatory to turn on video. 
• We will be giving up to 3 bonus points for participation during presentations (i.e., asking questions).

Review Format

We ask to prepare your reviews following the NeurIPS review guidelines  (see the section "Review Content"). In particular, your review should contain the following elements:

• Summary and contributions: Briefly summarize the paper and its contributions
• Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution.
• Weaknesses: Explain the limitations of this work along the same axes as above.
• Correctness: Are the claims and method correct? Is the empirical methodology correct?
• Clarity: Is the paper well written?
• Relation to prior work: Is it clearly discussed how this work differs from previous contributions?
• Reproducibility: Are there enough details to reproduce the major results of this work?
• Additional feedback, comments, suggestions for improvement and questions for the authors
• What do you see as the broader impact of this work, including potential negative ethical and societal implications of the work?
• Does the submission raise potential ethical concerns? This includes methods, applications, or data that create or reinforce unfair bias or that have a primary purpose of harm or injury. If so, please explain briefly.

For examples of reviews, have a look at the NeurIPS web page linked above.

You may submit the paper review as a team, including with the same team as for your presentation.

Late Submissions

You have 4 late days which you can use at any time during the term without penalty (for both assignments and projects). The final project writeup cannot be submitted late because we need to grade it in a short amount of time. Once you run out of late days, you will incur in a 20% penalty for each extra late day you use. When submitting as a team, each one of you must use a late day. Each late submission should be clearly marked as “Late” on the first page. No submission will be accepted 3 days after the deadline.

Collaboration Policy and Honor Code

You are free to form study groups and discuss homeworks and projects. However, you must write up homeworks and code from scratch independently without referring to any notes from the joint session. You should not copy, refer to, or look at the solutions in preparing their answers from previous years’ homeworks. It is an honor code violation to intentionally refer to a previous year’s solutions, either official or written up by another student. Anybody violating the honor code will be referred to the Office of Judicial Affairs.