Overview

• Instructor: Ferdinando Fioretto [email]
• Location: FALK ROOM 104
• Online class: Zoom link (must be logged with SYR account)
• Time: Mondays and Wednesday: 5:15-6:35pm
• Office hours: Mondays 9:00-10:00am (on Zoom)
• Deadline to choose your project: March 1
• Initial Project Report : March 22
• Project Progress Report: May 10
• Wellness days: Tuesday, March 23, and Wednesday, April 21.
• Class starts/ends: February 8, May 12
• Syllabus below
• Project Examples below

Schedule and material

Below is the calendar for this semester course. This is the preliminary schedule, which will be altered as the semester progresses. I will attempt to announce any change to the class, but this webpage should be viewed as authoritative. If you have any questions, please contact me.

Module 1: Evasion Attacks

Date	Topic	Reading	Presenter
Feb 8	Overview & motivation	slides | video
Feb 10	Attacks	C. Szegedy et al."Intriguing properties of neural networks" Optional Reading: A. Kurakin, I. Goodfellow, S. Bengio. "Adversarial examples in the physical world". N. Papernot, P. McDaniel, I. Goodfellow "Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples".	Fioretto
Feb 15	Attacks	D Lowd, C. Meek. "Good Word Attacks on Statistical Spam Filters" Optional Reading: N Dalvi et al.. "Adversarial classification".	Naveed
Feb 17	Attacks and Adversarial Training	I Goodfellow et al.."Explaining and harnessing adversarial examples" Optional Reading: F Tramèr, A Kurakin, N Papernot, I Goodfellow, D Boneh, P McDanie. "Ensemble Adversarial Training: Attacks and Defenses" B. Biggio et al.. "Evasion Attacks against Machine Learning at Test Time". Tutorial software: Cleverhans mnist tutorial (Pytorch)	Qi
Feb 22	Defensive Distillation	A Athalye, N Carlini, D Wagner. "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples" Optional Reading: Siyue Wang et al. "Defensive Dropout for Hardening Deep Neural Networks under Adversarial Attacks"	Jaimin
Feb 24	Defensive Distillation	N Papernot et al.."Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks" Optional Reading: G Hinton, O Vinyals, J Dean."Distilling the knowledge in a neural networks" J Jo, Y Bengio"Measuring the tendency of CNNs to Learn Surface Statistical Regularities" N Papernot et al."Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples"	Matthew

Module 2: Poisoning Attacks

Date	Topic	Reading	Presenter
Mar 1	Introduction	A Shafahi et al."Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks." Optional Reading: X. Huang et al."Is feature selection secure against training data poisoning?". B Nelson et al."Exploiting Machine Learning to Subvert Your Spam Filter."	Fioretto
Mar 3	Attacks on ML systems	"Poisoning attacks against support vector machines". Optimal Reading: M Jagielski et al."Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning". C Yudong, C Caramanis, S Mannor."Robust sparse regression under adversarial corruption".	Kai
Mar 8	Defense Mechanisms	B Rubinstein et al."ANTIDOTE: Understanding and Defending against Poisoning of Anomaly Detectors". Optional Reading: J Steinhardt, PW Koh, P Liang."Certified Defenses for Data Poisoning Attack". B Wang et al."Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks".	Sawinder

Module 3: Privacy Attacks

Date	Topic	Reading	Presenter
Mar 10	Data Exposure	A Narayanan, V Shmatikov."Robust de-anonymization of large sparse datasets.". Y de Montjoye et al.Unique in the Crowd: The privacy bounds of human mobility Optional Reading: Y de Montjoye et al.Unique in the shopping mall: On the reidentifiability of credit card metadata F. Tramèr et al."Stealing Machine Learning Models via Prediction APIs". C Dwork et al."Exposed! A Survey of Attacks on Private Data". JA Calandrino et al."‘You Might Also Like’: Privacy Risks of Collaborative Filtering".	Borui
Mar 15	Privacy Attacks in Deep Learning	R Shokri et al."Membership Inference Attacks against Machine Learning Models". OptionalReading: M Nasr, R Shokri, A Houmansadr. "Comprehensive Privacy Analysis of Deep Learning".	My
Mar 17	Privacy Attacks in Deep Learning	N Carlini et al."The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. ". Optional Reading: C Song, V Shmatikov."Overlearning Reveals Sensitive Attributes.".	Ambarish
Mar 22	Initial Project Report	Everybody presents

Module 4: Foundations of Differential Privacy

Date	Topic	Reading	Presenter
Mar 24	Preliminaries	C Dwork, A Roth."The Algorithmic Foundations of Differential Privacy". Chapters 2 and 3	Fioretto
Mar 24	Exponential Mechanism and Composition	C Dwork, A Roth."The Algorithmic Foundations of Differential Privacy". Chapter 3 Kairouz et al.:The Composition Theorem for Differential Privacy	Fioretto
Mar 29	Data Generation	J Zhang et al."PrivBayes: Private Data Release via Bayesian Networks". Optional Reading: G. Cormode et al. " Differentially private spatial decompositions ". C Liand, G Miklau. " An adaptive mechanism for accurate query answering under differential privacy ".	Ravi
Mar 31	Optimization	F Fioretto P Van Hentenryck.:"Differential Privacy of Hierarchical Census Data: An Optimization Approach". Optional Reading: H Rastogi, M Suciu.:  " Boosting the Accuracy of Differentially Private Histograms Through Consistency ". T Mak, F Fioretto, P Van Hentenryck.:  " Privacy-Preserving Obfuscation for Distributed Power Systems ".	Prithvi

Module 5: Differential Privacy and Machine Learning

Date	Topic	Reading	Presenter
Apr 5	DP Empirical Risk Minimization	C Dwork, A Roth:"The Algorithmic Foundations of Differential Privacy". Chapter 11 K Chaudhuri et al.:"Differentially private empirical risk minimization ".	Fioretto
Apr 7	DP Stochastic Gradient Descent	M Abadi et al."Deep learning with differential privacy". Balle et al."Privacy amplification by subsampling: tight analyses via couplings and divergences".	Fioretto
Apr 12	Other Deep Learning Approaches	N Papernot et al."Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data". N Papernot et al."Scalable Private Learning with PATE". Software: cleverhans TensorFlow privacy	TBD
Apr 14	Generative Adversarial Networks	L Xie et al.: "https://arxiv.org/abs/1802.06739". Optional Reading: Gergely Acs et al.:  " Differentially Private Mixture of Generative Neural Networks ". X Zhang, S Ji, T Wang.:  " Differentially Private Releasing via Deep Generative Model (Technical Report) ". C Arnold, M Neunhoeffer, S Sternberg:  " An Empirical Evaluation of Differentially Private Generative Adversarial Networks for Science ".	TBD

Module 6: Differential Privacy Model Extensions

Date	Topic	Reading	Presenter
Apr 19	Local DP	Local Privacy and Statistical Minimax Rates. Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity. Optional Reading P Kairouz, S Oh, P Viswanath.:  " Extremal Mechanisms for Local Differential Privacy ". B Bebensee.:  " Local Differential Privacy: a tutorial ".	Fioretto
Apr 21	Wellness day
Apr 26	Temporal DP	F Fioretto, P Van Hentenryck.:  " OptStream: Releasing Time Series Privately ". Optional Reading: V Rastogi, S Nath. :  " Differentially private aggregation of distributed time-series with transformation and encryption ". Y Cao et al: "Quantifying Differential Privacy in ContinuousData Release under Temporal Correlations".	TBD
Apr 26	Deployments	U Erlingsson, V Pihur, A Korolova.:"RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response". Abowd, Ashmead, Garfinkel, et.al., 2019. Census TopDown: Differentially Private Data, Incremental Schemas, and Consistency with Public Knowledge Optional Reading: B Ding, J Kulkarni, S Yekhanin:  " Collecting telemetry data privately ". Kifer, Messing, Roth, Thakurta, Zhang, 2020. Guidelines for Implementing and Auditing Differentially Private Systems Apple Differential Privacy Team, 2017. Learning with Privacy at Scale and Differential Privacy Overview Kenthapadi and Tran. SIGKM 2018. PriPeARL: A Framework for Privacy-Preserving Analytics and Reporting at LinkedIn	TBD

Module 7: Federated Learning

Date	Topic	Reading	Presenter
May 1	Preliminaries	McMahan et al.Communication-Efficient Learning of Deep Networks from Decentralized Data Optional Reading: Kairouz et al.: "Advances and Open Problems in Federated Learning"	TBD
May 3	Privacy	McMahanLearning differentially private recurrent language models Optional Reading: Geyer et al.Differentially private federated learning: a client level perspective Liang et al.: Think Locally, Act Globally: Federated Learning with Local and Global Representations.	TBD

Final Presentation

Date	Topic	Reading	Presenter
May 10	Poster Session
May 12	Poster Session

Syllabus

Assignments

Paper presentation

In each class, a team of students will present the assigned papers. Different types of presentation are allowed (e.g., slides, interactive demos or code tutorials). The only requirements is that the presentation should (a) involve the class in active discussions, (b) cover all papers assigned for reading, and (c) last no more than 1:15h including discussions.

Research projects

Students will work on a course-long research project. Each project will be presented on May 10 or 12.

Grading

Grading scheme

50% paper presentation, 10% class participation, 40% research project.
Paper presentations will be graded according to the this rubric.

Class participation

Course lectures will be driven by the contents of assigned papers. All students are asked to participate in an active discussions of the paper content during each class.
During the class each student is required to as at least ONE question

Lateness policy

The presentation material should be presented two days prior the day of presentation. A 10% per-day late-penalty will be applied for delays. If the presentation is not ready for the day in which the team is supposed to present all students in the team will be assigned 0 points.

Ethics statement

In this course, you will be learning about and exploring some vulnerabilities that could be exploited to compromise deployed systems. You are trusted to behave responsibility and ethically. You may not attack any system without permission of its owners, and may not use anything you learn in this class for evil. If you have doubts about ethical and legal aspects of what you want to do, you should check with the course instructor before proceeding. Any activity outside the letter or spirit of these guidelines will be reported to the proper authorities and may result in dismissal from the class.

Integrity

Syracuse University’s Academic Integrity Policy reflects the high value that we, as a university community, place on honesty in academic work. The policy defines our expectations for academic honesty and holds students accountable for the integrity of all work they submit. Students should understand that it is their responsibility to learn about course-specific expectations, as well as about university-wide academic integrity expectations. The policy governs appropriate citation and use of sources, the integrity of work submitted in exams and assignments, and the veracity of signatures on attendance sheets and other verification of participation in class activities. The policy also prohibits students from submitting the same work in more than one class without receiving written authorization in advance from both instructors. Under the policy, students found in violation are subject to grade sanctions determined by the course instructor and non-grade sanctions determined by the School or College where the course is offered as described in the Violation and Sanction Classification Rubric. Syracuse University students are required to read an online summary of the University’s academic integrity expectations and provide an electronic signature agreeing to abide by them twice a year during pre-term check- in on MySlice.
Any instance of sharing or plagiarism, copying, cheating, or other disallowed behavior will constitute a breach of ethics. Students are responsible for reporting any violation of these rules by other students, and failure to constitutes an ethical violation that carries with it similar penalties.
The Violation and Sanction Classification Rubric establishes recommended guidelines for the determination of grade penalties by faculty and instructors, while also giving them discretion to select the grade penalty they believe most suitable, including course failure, regardless of violation level. Any established violation in this course may result in course failure regardless of violation level.

Academic Integrity Online

All academic integrity expectations that apply to in-person quizzes and exams also apply to online quizzes and exams. In this course, all work submitted for quizzes and exams must be yours alone. Discussing quiz or exam questions with anyone during the quiz or exam period violates academic integrity expectations for this course.

Stay Safe Pledge

As part of the university’s plan for re-opening, all students are expected to affirm their commitment to keeping themselves and the campus community safe by signing the Stay Safe Pledge. The Pledge requires students to wear a mask or face covering while on campus, maintain six feet of distance from others, and avoid attending class or participating in campus activities when feeling unwell. Instructors will enforce these expectations in their classrooms. Further guidance, including tips on how to address students who are not upholding these requirements, may be found in The Stay Safe Pledge: Guidance for Faculty, TAs, and Instructional Staff. The following language should be included prominently in your syllabus and highlighted at your first class session:
Syracuse University’s Stay Safe Pledge reflects the high value that we, as a university community, place on the well-being of our community members. This pledge defines norms for behavior that will promote community health and wellbeing. Classroom expectations include the following: wearing a mask that covers the nose and mouth at all times, maintaining a distance of six feet from others, and staying away from class if you feel unwell. Students who do not follow these norms will not be allowed to continue in face-to-face classes; repeated violations will be treated as violations of the Code of Student Conduct and may result in disciplinary action.

Food and Drink in the Classroom

Eating and drinking require the lowering of the face mask, creating a potentially dangerous situation. For this reason, students are not allowed to eat or drink in class during the COVID-19 pandemic. Instructors teaching classes that are longer than 80 minutes in duration should allow students to leave the room as needed or include a short break to allow students to get a drink.

Online Etiquette

Students participating remotely in hybrid and synchronous online class sessions can be expected to conduct themselves as they would in an in-person class. They should dress and behave as they would in a face-to-face class. The issue of whether or not students must keep their web cameras on during class is complicated. In general, it is reasonable to expect students to keep their cameras on. However, faculty should be sensitive to each student’s personal circumstances and be prepared to find an equitable solution in cases where students are uncomfortable keeping their cameras on during class.
In both hybrid and fully online classes, students should use the “raise hand” function to ask questions and refrain from interrupting the class. Faculty may wish to download and review the “chat” transcript from each class session. You may wish to review, or refer your students to, the ‘Netiquette for Students’ resource at the ITS Answers page.

Use of Class Materials and Recordings

Original class materials (handouts, assignments, tests, etc.) and recordings of class sessions are the intellectual property of the course instructor. You may download these materials for your use in this class. However, you may not provide these materials to other parties (e.g., web sites, social media, other students) without permission. Doing so is a violation of intellectual property law and of the student code of conduct.

Links to Related Courses

• Trustwothy Machine Learning
• Security and Privacy of ML