This course takes a hands-on approach to understanding machine learning systems by having students build their own ML framework from the ground up. Rather than treating ML libraries as black boxes, students will implement core components—from auto-differentiation engines to GPU kernels—gaining deep insight into the engineering challenges of making ML systems correct, robust, and fast. Through this implementation-focused approach, the course covers essential topics including tensor operations, model debugging and visualization, GPU fundamentals, and optimization techniques for deployment.

 

Instructor: Ramtin Keramati, PhD

Class Location: Bloomberg Center 131

Time: Mon - Wed (7:30PM - 8:45PM)

Credits: 3

Course materials are kindly shared by Prof. Alexander (Sasha) Rush. 

 

Course Structure 

CS 5781 is for students who want to master the engineering side of ML systems. Rather than surveying a wide range of tasks and algorithms, the course follows the full pipeline of implementing, optimizing, and deploying a single model. By focusing on a fixed architecture, we can dive deeply into each stage—from low-level hardware considerations to high-level user tools—and explore the trade-offs in training, debugging, testing, and deployment.

The course is less math-heavy than most ML classes, requiring only basic familiarity with matrices and derivatives, but it is highly programming-intensive. Assignments involve substantial Python coding, culminating in a complete, working ML system.

This year’s focus is on building and optimizing a compact deep-learning model (image recognition system). The semester is organized into:

• Module 0: Fundamentals
• Module 1: Autodifferentiation
• Module 2: Tensors
• Module 3: Efficiency
• Module 4: Networks

 

Office Hours

• Ramtin Keramati -- Tuesday 5:00pm - 6:00pm (Zoom Link)
• Ahmed Abouelhamayed --- Thursdays 2:30-4:00 -- BLM398
• Jorge Tapias Gomez -- Wednesdays 2:00-3:30pm -- BLM375
• Jake Hyun -- Mondays 10:00-11:30 am -- BLM398

 

Grading

• 60% Module Homework Assignments (late submissions: –10% per day)

• 20% In-Class Mid Term 

  • Wednesday Nov 5th

• 10% In-Class Quiz Grades (based on accuracy)

• 10% Attendance (credit given for being present and submitting the quiz)

Assessment and Deliverables

Assessment is entirely problem-set and quiz-based, with 5 assignments released approximately every two weeks. While students may collaborate and discuss approaches, each person must submit their own individual implementation.

Every student builds their own complete ML framework from scratch. Each assignment adds a key component, and by semester's end, you'll have a fully functional deep learning library capable of training modern neural networks efficiently.

Assignment Sequence:

• Assignment 0: Testing, Modules, and Visualization
• Assignment 1: Auto-Derivatives and Training
• Assignment 2: All about Tensors
• Assignment 3: Speeding Things Up
• Assignment 4: Building Real Models

Assessment focuses primarily on programming assignments that develop ML systems. A few written questions will ask you to explain your reasoning and engineering decisions. Grading emphasizes code accuracy through automated tests and model performance, with written components graded for completeness.

Learning Outcomes

Understand How AI Really Works: Gain deep, fundamental knowledge of how neural networks actually compute and learn—not just how to use them as black boxes.

Master Mathematical Programming: Develop the ability to translate mathematical concepts from calculus and linear algebra into working code that solves real problems.

Build High-Performance Systems: Learn to write efficient, optimized code that handles large-scale computations and leverages modern hardware like GPUs.

Develop Framework Design Skills: Understand how to architect complex software systems with proper abstractions, modularity, and extensibility—skills that transfer beyond ML.

Gain Implementation Confidence: Develop the skills to implement any machine learning algorithm from scratch, making you self-sufficient in understanding and creating AI systems.

 

Preparation

Programming: CS 2110 or equivalent programming experience required.

Mathematics: Comfort with calculus and probability is essential, particularly differentiation and basic discrete distributions. No proofs or extensive symbolic manipulation required.

Computer Science: This is a programming-intensive course. You should have strong Python proficiency and familiarity with at least one numerical library (numpy, scipy, scikit-learn, PyTorch, or TensorFlow). Understanding of foundational CS concepts—memory management, computational complexity, and algorithm analysis—is expected.

 

Topics Covered

• Implementation of backpropagation engines
• Low-level GPU programming and hardware optimization
• High-level autodifferentiation in modern libraries
• Core models for vision, text, and recommendation systems
• Debugging and testing large-scale machine learning systems
• Model visualization, hyperparameter tuning, and deployment strategies
• Integration with data processing pipelines

 

FAQ

What resources do I need to complete the class?

• The course will require you to have a python development environment set up, ideally on your own machine or on a cloud server. It does not need to be very powerful nor will that help you do better in the class. We will have some lectures using GPUs, but will use Google Colab for these lectures.  

What technologies do I need to know to complete the class?

• This is a software engineering style course, and so we recommend that you have a strong background in standard tools such as Git and GitHub, Python, and command-line programming. 

What math do I need to know to complete the class?

• This course will require light-undergraduate level calculus and vector manipulation. We will provide resources for reviewing these aspects in homework assignments. 

 

Academic Integrity

All students should abide by the Cornell University Code of Academic Integrity, and all writing submitted should be one’s own writing. While discussing course concepts with other students is highly encouraged, plagiarism (including use of ChatGPT, ...) will result in zero credit and/or a referral to the Office of Student & Academic Affairs. Please reach out if additional questions arise on what is or is not permitted.

 

Students with Disabilities

Your access in this course is important to us. Please give us your Student Disability Services (SDS) accommodation letter early in the semester so that we have adequate time to arrange your approved academic accommodations. If you need immediate accommodations for equal access, please speak with us after class or send an email message to us and/or SDS at sds_cu@cornell.edu. If the need arises for additional accommodations during the semester, please contact SDS. You may also feel free to speak with the Student & Academic Affairs team at Cornell Tech who will connect you with the university SDS office. If you have, or think you may have a disability, please contact Student Disability Services for a confidential discussion: sds_cu@cornell.edu, 607-254-4545, sds.cornell.edu. You must request your SDS accommodation letter no later than 3 weeks prior to needing it.

• Students currently registered with SDS: Once you request your accommodation letter and it is approved by SDS, it will be emailed to both you and your instructors. Processing time can be up to 48-hours. 
• Students not registered with SDS: The registration process for new accommodations can take up to three weeks. Once you are approved by SDS for accommodations, you will be able to request your accommodation letter for this course.
• If you are approved for accommodations later in the semester: you must request your accommodation letter as soon as possible.

 

Mental Health & Well-being 

Your health and wellbeing are important to us, and you should always feel free to reach out to us for support. There are services and resources at Cornell designed specifically to bolster student mental health and well-being. Remember, your mental health and emotional well-being are just as important as your physical health. If you or a friend are struggling emotionally or feeling stressed, fatigued, or burned out, there are many campus resources available to you:

This link provides a list of resources for Cornell Tech students: https://mentalhealth.cornell.edu/get-support/tech. You can additionally also contact studentwellness@tech.cornell.edu with concerns.

 

Religious Observances

Cornell University is committed to supporting students who wish to practice their religious beliefs. Students are advised to discuss religious absences with their instructors well in advance of the religious holiday so that arrangements for making up work can be resolved before the absence. Students are encouraged to anticipate their religious/spiritual needs early in the semester, and at least two weeks before the observance, leaving plenty of time for the professor and student to reach a reasonable accommodation.