Generative models are a class of machine learning algorithms that operate over complex, high-dimensional objects such as images, sequences, and graphs. Recent advances have greatly improved the capabilities of generative models and have enabled new applications in computer-generated art, natural language processing, computational drug design, and other areas. This course explores the foundational probabilistic principles of generative models, their learning algorithms, and popular model families, which include variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, and diffusion models. The course also covers applications in domains such as computer vision, natural language processing, and biomedicine, and draws connections to the field of reinforcement learning.

First Lecture Information

The first lecture will be held on Monday 01/22 in Bloomberg 91 (streamed to Gates G11). The following Zoom link will also be available to connect to the lecture:

• Zoom link:   https://cornell.zoom.us/j/97826794797?pwd=WUd6aUNrM1dEeDhRMWJWNGNqYVVFZz09 

You may also find Zoom links to other lectures in Canvas under the "Zoom" tab.

Lecture recordings from last year are available at: https://www.youtube.com/playlist?list=PL2UML_KCiC0UPzjW9BjO-IW6dqliu9O4B 

Key Dates

Please note the following important dates for this semester:

• January 29: Assignment 1 is released
• February 12: Assignment 1 is due
• February 14: Assignment 2 is released
• February 21: Project proposal is due
• February 27: No class (February break)
• March 4: Assignment 2 is due.
• March 4: Assignment 3 is released
• March 18: Assignment 3 is due
• April 10: Assignment 4 is released
• April 24: Assignment 4 is due
• March 25 to April 26: Student presentations
• March 31: Project progress report is due
• April 1 & 3: No class (spring break)
• April 10: Assignment 4 is released
• April 24: Assignment 4 is due
• May 1 to May 9: Time off to work on projects
• May 12: Final project report is due

Please use this link to sign up for presentations: https://docs.google.com/spreadsheets/d/1RbjnwVAIIWjVVA9ZZgE6Q6nRqQHJlXEiD5RYE6GzFDk/edit?usp=sharing

Information and Instruction Format

Instructor: Volodymyr Kuleshov

Credits: 3

Times: Mon/Wed - 2:55-4:10pm Eastern Time.

Location: Bloomberg 91 (Tech Campus). Ithaca campus room is Gates G11.

The course will be offered in a hybrid in-person/remote format:

• The instructor will give the first several weeks of lectures in person from the Tech campus (Bloomberg 91). The lectures will be streamed to an Ithaca classroom (Gates G11) as well as on Zoom.
• Because of conference travel, several in-person lectures will be held remotely. Students will be notified ahead of time via Canvas.
• The last third of the course will consist of student presentations. Students can form teams across the two campuses and present over Zoom. We do not anticipate using the classrooms at that point, unless the students prefer to give their presentation from a classroom.

Teaching Staff and Office Hours

Volodymyr Kuleshov (Instructor). https://www.cs.cornell.edu/~kuleshov/

Office Hours: MW after class. 4:10-4:30pm. 

Prerequisites

• Basic knowledge about machine learning from at least one of: CS4780, CS4701, CS5785.
• Basic knowledge of probability and calculus: students will work with computational and mathematical models.
• Basic knowledge of deep neural networks (CNNs, RNNs; CS5787). Experience implementing deep neural networks is not required but will be helpful for the class project.
• Proficiency in a programming language (preferably Python).

Grading

Assignments

The course will feature three assignments/homeworks that will contain a mix of written and programming questions. Assignments will be released, submitted, and graded using Gradescope.

If you're not already in Gradescope, you can use this code to enroll: [TBD]

The programming questions should be done in PyTorch using the provided starter code. You will be required to submit figures generated using your code as a separate PDF in Gradescope.

The written part of the homeworks should be written up clearly and succinctly; you may lose points if your answers are unclear or unnecessarily complicated. You are encouraged to use LaTeX to writeup your homeworks, but this is not a requirement. You can find a basic LaTeX template here. Assignments will be submitted on Gradescope.

You are encouraged to work in teams of two: make sure to put both of your names on the submission and submit as a team in Gradescope.

Course Project

The course project will give the students the chance to explore generative models in greater detail. Course projects will be done in groups of up to 3 students and can fall into one or more of the following categories:

• Application of machine learning to a practical problem or a dataset.
  
• Improvements to machine learning algorithms.
  
• Theoretical analysis of any aspect of machine learning models.
  

Pick a topic that's meaningful to you and that excites you. For example, if you do PhD research in biology, you can do a project related a dataset that you work with as part of your thesis research. If you're an Urban Tech student, you can work with an urban dataset that you find interesting. You are encouraged to find something on your own, but feel free to talk to the teaching team during office hours about project ideas.

Proposal

Your proposal should give the title of the project, the project category, the names of your team members, their NetID, and a 300-500 word description of what you plan to do. It should contain the following information.

• Motivation: What problem are you tackling? Is this an application or a theoretical result?
• Method: What machine learning techniques are you planning to apply or improve upon and how?
• Future work: What experiments are you planning to perform or what theorems do you want to prove?

The goal of the proposal is make sure you're on the right track. As long as you follow the above guidelines, you should do well.

Please submit the proposal via Gradescope and make sure to submit as a team.

Milestone 

The milestone submission should describe what you've accomplished so far, and briefly say what else you plan to do. The format should be the same as of the final project, with an approximate length of 3-5 pages (excluding references). The goal is to make sure that you are on track to finish the final project.

• Motivation: What problem are you tackling? Is this an application or a theoretical result?
• Method: What machine learning techniques are you planning to apply or improve upon and how?
• Preliminary experiments: Describe the experiments that you've run, the outcomes, and any error analysis that you've done. You should have tried at least one baseline.
• Future work: What else do you plan to do?

The goal of the milestone is make sure you're on the right track. As long as you follow the above guidelines, you should do well.

Please submit the milestone via Gradescope and make sure to submit as a team.

Final Writeup

The final writeup should describe all the work you did for your course project and summarize the main results. You can think of it as a technical report that presents your findings to a general machine learning audience.

The style and format of the writeup should be similar to that of a research paper. The maximum length is 8 pages, excluding references. 

There are no strict requirements on the structure of the final writeup, but one way to structuring it would be include the following sections, which are fairly standard for a research paper.

• Abstract: Summarize the problem, novel contributions, and results in one paragraph.
• Introduction: Provide motivation for the problem and expand upon the overview in the abstract.
• Background: Briefly summarize the background knowledge needed to understand the work.
• Method: Describe the methods that will be used or implemented in the paper.
• Theoretical analysis: If you are doing a theory project, describe your theoretical results here.
• Experimental analysis: Describe in detail your experiments.
• Discussion and Prior Work: Discuss the key takeaways from your experiments. Put your results in the context of previous work
• Conclusion. You may summarize the paper or talk about open problems and open directions.

Regardless of how the writeup is structured, please make sure to cover the following points.

• Motivation: What problem are you tackling? Why is it interesting? What type of project will this be (application, method, theory)?
• Method: What machine learning techniques are you planning to apply or improve upon and how? Make sure to describe them in detail and provide enough context for the reader to understand the methods at least at a high level. Provide any background that is necessary for that.
• Experiments: Describe the experiments that you've run, the outcomes, and any error analysis that you've done. Make sure that the setup is described in enough detail for someone else to reproduce your results. Also, if you have an experimental project, make sure to provide a detailed experimental analysis. Things you should consider including are: train/test performance, learning curves, model samples, error analyses, ablation analyses, etc. Most projects should also include baselines.
• Theory: If doing a theory project, state your results formally as theorems. Make sure that all the symbols are defined. Also, the best presentation of theoretical results tends to also explain the results in plain language and conveys the intuition behind them.
• Context: Explain how you build upon previous work and how your results compare to what has been done previously.

Writeups will be evaluated for their presentation clarity, the respect of the above guidelines, the significance of the project (does it explore a toy dataset or a real problem) and the technical quality of the work (the level of depth in the experimental or theoretical analyses, does the approach make sense technically, are the algorithms implemented reasonable and studied in enough detail, etc.).

Please submit the writeup via Gradescope and make sure to submit as a team.

Presentations

Students will be asked to deliver a presentation in the second half of the course. We will follow the role-playing presentation format described here: https://canvas.cornell.edu/courses/63240/pages/presentation-instructions

Each student is required to choose a paper from the official paper list. If students have a strong preference to choose a paper not on the list, they should reach out to the instructor in advance and argue for their proposed choice of paper.

https://canvas.cornell.edu/courses/63240/pages/paper-list 

Presentations will be done in teams of 1-2, and each student presents once. Additionally, each student is required to sign up for a supporting role in every two-lecture block. Teams sizes for supporting roles are 1-2 students/role.

The presentation component of the course consists of two components. Half of the grade will be assigned to the main presentation. The other half of the grade will be assigned to participation in supporting role. After each class, each team that performed in a supporting role will get a grade, and the final grade for the supporting role will be the average of these per-presentation grades.

Here are some additional guidelines.

• Have a look at the papers and roles and sign up for presentations and roles on the spreadsheet. 
• We will be asking for mandatory attendance to the presentations. It is also mandatory to turn on video. 

Late Submissions

You have 4 late days which you can use at any time during the term without penalty (for both assignments and projects). The final project writeup cannot be submitted late because we need to grade it in a short amount of time. Once you run out of late days, you will incur in a 20% penalty for each extra late day you use. When submitting as a team, each one of you must use a late day. Each late submission should be clearly marked as “Late” on the first page. No submission will be accepted 3 days after the deadline.

Collaboration Policy and Honor Code

You are free to form study groups and discuss homeworks and projects. However, you must write up homeworks and code from scratch independently without referring to any notes from the joint session. You should not copy, refer to, or look at the solutions in preparing their answers from previous years’ homeworks. It is an honor code violation to intentionally refer to a previous year’s solutions, either official or written up by another student. Anybody violating the honor code will be referred to the Office of Judicial Affairs.