Statistics 234
Spring,
2027
This
graduate course will focus on reinforcement learning algorithms and sequential
decision-making methods with special attention to how these methods can be used
in digital health. Reinforcement learning (RL) is the area of machine learning
which is concerned with sequential decision making. We will focus on the areas
of sequential decision making that concern both how to select optimal treatment
actions as well as how to evaluate the impact of these actions.
Digital health
is an area that lies within multiple scientific disciplines including
statistical science, computer science, behavioral science, and cognitive
neuroscience. This makes for very exciting interdisciplinary work! Smartphones
and wearable devices have remarkable sensing capabilities allowing us to
understand the context in which a person is at a given moment. These devices
also can deliver treatment actions tailored to the specific needs of users in a
given location at a given time. Figuring out when and in which context, which
treatment actions to deliver, can assist people in achieving their longer-term
health goals. Across multiple classes we will brainstorm about how the methods
we discussed during that class might be useful in digital health.
This course
will cover the following topics: Markov Decision Processes, on-policy and
off-policy RL, least squares methods in RL and Bayesian RL, namely posterior
sampling. Most of the course will focus on Bayesian RL via posterior sampling. Bayesian
RL is particularly useful in digital health as posterior sampling facilitates
off-policy and continual learning. Further the Bayesian paradigm facilitates
the use of prior data in initializing an RL algorithm. At the end of the
semester we will work through a TBN theoretical paper in RL. Other topics from
statistics, machine learning and RL that are rather interesting but that we may
or may not cover are (you could consider in your class project) include: 1)
transfer learning (using data on other similar users/tasks to enable faster
learning); 2) methods for warm-starting online RL algorithms 3)
interpretability of policies (enabling communication with behavioral scientists
by making connections to behavioral theories); 4) using RL to train LLMs, 5)
multi-agent RL and 6) using RL to determine prompts that guide an LLM.
Professor: Susan Murphy (samurphy@g.harvard.edu).
Classroom and Times: Maxwell Dworkin ???: Monday and Wednesday 4:30pm-5:45pm. No class 2/15, 3/15, 3/17.
TF:
Office
Hours:
Susan's Office Hours: SEC 2.335, by
appointment only, at 9:30am on Thursdays. NO OFFICE HOURS 3/18
xxx's Office Hours: xxx
xxx's Office Hours:xxx
Book: Sutton R. & Barto A. (2020). Reinforcement Learning:
An Introduction (2nd Edition). Cambridge: The MIT Press. No
purchase is necessary; you can download a pdf copy here.
Ch. 21 of Russell & Norvig,
(Artificial Intelligence A Modern Approach, 3rd edition is on Canvas in the
Files Section). 4th edition can be found via the Harvard Library Hollis.
Required
Papers: A variety of papers will be assigned;
see below.
Prerequisites: Recommended
prerequisites are the equivalent of stat210 and compsci181.
Typical
Class:
4:30pm: Sit
with your breakout group.
4:30-4:45pm:
Quiz on 1 randomly selected prior assigned question and assigned reading
material
4:45-5:15pm: Discuss the x previously
assigned questions + new question
5:15-5:25pm: Breakout with your group
(Discuss new question)
5:25-5:35pm: Class Discussion (Randomly
selected group leads the discussion)
5:35-5:45pm: Pointers for investigating
the x new questions
Course
Outline: This outline will be constantly updated please
check prior to each class!
|
Date |
Topic |
Reading
Assignments |
|
|
01/25 |
Intro |
||
|
01/27 |
Intro |
OptimalPolicyStationary.pdf
On Canvas under Files> Discussions and Other Helpful Material |
|
|
02/01 |
Bandit |
Ch.
2 of Sutton &
Barto |
|
|
02/03 |
|||
|
02/08 |
Bandit |
Ch.
2 of Sutton &
Barto |
|
|
02/10 |
MDPs |
Ch. 3
of Sutton &
Barto |
|
|
02/17 |
MDPs.
Two decision making problems: The learning algorithm (Bandit alg./RL
algorithm) and the policy that solves MDP |
Ch. 4
of Sutton &
Barto TemporalCreditExplorationExploitationDiscussion.pdf
This file is on Canvas under Files> Discussions and Other Helpful Material |
|
|
02/22 |
MDPs |
Ch. 5
of Sutton &
Barto Files
on Canvas: OptimalPolicyStationary.pdf and M_Z_Estimating Functions.pdf |
|
|
02/24 |
TD
Learning & Control |
Ch. 6
of Sutton & Barto. TemporalCreditExplorationExploitationDiscussion.pdf
On Canvas under Files> Discussions and Other Helpful Material |
|
|
Each student must arrange a 30 min. meeting with xxx or xxx or Susan to
discuss initial project ideas between 02/22-03/03 |
|||
|
03/01 |
|||
|
03/03 |
TD
Learning |
Ch. 6
of Sutton & Barto. EligibilityTracesDiscussion.pdf
and M_Z_Estimating
Functions.pdf On
Canvas under Files> Discussions and Other Helpful Material |
|
|
03/08 |
Control Least
Squares Methods in RL |
Sections
6.4-6.6 of Sutton & Barto. |
|
|
03/10 |
Control Least
Squares Methods in RL |
Sections
6.4-6.6 of Sutton & Barto. The file
Discount_Factor.pdf which is on Canvas under Files> Discussions and Other
Helpful Material |
|
|
**Initial Project Proposal Due in Canvas 03/23, 5pmEST**; see the STAT
234 Project Evaluation in Canvas Files. |
|||
|
03/22 |
Batch,
Off-Policy RL |
The
file Discount_Factor.pdf which is on Canvas under Files> Discussions and
Other Helpful Material Read Review of Batch RL |
|
|
03/24 |
|||
|
**Revised Project Proposal Due in Canvas on 03/30, 5pmEST**; see the STAT
234 Project Evaluation in Canvas Files |
|||
|
03/29 |
|||
|
03/31 |
LSPI,
More Batch, Off-Policy RL |
LSPI, LSVI (only need to understand
algorithm 3 in appendix) M_Z_Estimating
Functions.pdf On
Canvas under Files> Discussions and Other Helpful Material |
|
|
04/05 |
LSVI,
Thompson Sampling |
LSVI (only need to understand
algorithm 3 in appendix) Tutorial on Thompson-Sampling,
(Sections 1-4) File on
Canvas: LSVI_Notes.pdf |
|
|
04/07 |
|
||
|
04/12 |
Thompson
Sampling. Connect to L_2 penalization |
Russo, Van Roy, Kazerouni, Osband
and Wen, 2017, revised 2020 (Sections 1-4, 7.1,7.5,
8.1.3, example 8.4) |
|
|
04/14 |
Thompson
Sampling. Connect to L_2 penalization |
Russo, Van Roy, Kazerouni, Osband
and Wen, 2017, revised 2020 (Sections 1-4, 7.1,7.5,
8.1.3, example 8.4) |
|
|
04/19 |
|||
|
04/21 |
|||
|
04/26 |
|||
|
Posters Due in Canvas on 04/23 at 5pm EST; see the STAT 234 Project
Evaluation in Canvas Files |
|||
|
04/28 |
|
||
|
**Projects Due 05/05 in Canvas** 5pm EST; see the STAT 234 Project
Evaluation in Canvas Files |
|||
Grading: Course grades will be based on a weighted average of quizzes
(50%), participation (10%), final project (40%). Note that the participation
grade concerns showing up for class and participating in class discussions. The
40% credit for the project will be split as follows:
a.
Project proposal (5%)
b.
In-person presentation with ability to
answer questions between 04/21-04/28 (20%)
c.
Project final report (15%)
See the file STAT 234 Project
Evaluation in Canvas Files for how project grades are determined.
Quizzes: The quiz is about the assigned reading and one of the
randomly selected questions. Assigned Readings are provided above in the Course
Outline. To help with various circumstances (expected or unexpected), your
lowest three (3) quizzes will be dropped. Collaboration
on Quizzes is not permitted. Use of generative AI is not permitted.
Projects: An important component of the course is a final project which
can either be a survey of some actively developing sub-topic within sequential
decision making or a research project involving contributing novel research
(theoretical result, statistical method, computational algorithm) to the area
of sequential decision making. Example projects from prior years are here. Note that projects need not pertain to digital health. See the STAT 234 Project
Evaluation in Canvas Files.
Surveys must be written individually.
However, teams of up to 2 students can be formed for a research project. To get
full credit, surveys must be very high quality: they should be like a
publishable survey article in a top journal. The bar for research projects will
be lower because of the time constraint and the inherent uncertainty in the
research process. While you are not required to deliver publication quality
research work by the end of the semester, you are encouraged to do so. We will
provide some suggestions for research projects, but you should feel free to
work on ANY problem in sequential decision making that interests you. The
papers must be written according to the submission rules at ICML: https://icml.cc/Conferences/2026/AuthorInstructions.
It is easiest to use Latex with the style files ICML provides. These are 8-page papers.
Using generative AI tools such as
ChatGPT to help with your project is allowed. In my own testing of ChatGPT, it
often has good points but makes many mistakes. It could be a useful tool for
suggesting ideas (and to chat about the material with or compare answers with)
but it is error-prone (particularly concerning technical/mathematical material).
The references can be completely fictious. Furthermore, working hard on the
project is crucial for learning and to help you grow in independence. So even
if ChatGPT did excellent writing ICML-style papers, relying on it too much
would be harmful for you. In any case, your project, poster and presentation
must reflect your own understanding of the material, explained in your own way,
rather than being copied from any other source.
Posters:
During 04/21-04/28 you will show your poster and answer
questions about the content of the poster during class. Posters will be due in
Canvas at 5pm on Tuesday 4/23. An example poster can be found here. Here is a pptx format for the poster.
You should make 6 highly informative (non-dense) slides and display them in 2 x
3 or 3 x 2 format as in this pptx file. Other creative formats are also
welcome.
Breakout
Groups and Participation: Active participation is expected,
through attending class (Monday & Wednesdays), completing quizzes and
engaging in classroom discussions/breakout groups. Every two weeks you will be
assigned a new breakout group. The first breakout groups will be formed Monday
evening January 25, then Feb. 08, Feb 22, March 08, March 29 and April 12. Stat
234 covers subtle concepts, so let's all try to help create a supportive,
collaborative community.
Accommodations: Harvard University s goal is to remove barriers for disabled
students related to inaccessible elements of instruction or design in this
course. If reasonable accommodations are necessary to provide access, please
contact the Disability Access Office (DAO). Accommodations do not alter fundamental requirements of
the course and are not retroactive. Students should request accommodation as
early as possible, since they may take time to implement. Students should
notify DAO at any time during the semester if adjustments to their communicated
accommodation plan are needed.
Where to get tech help: The Academic
Resources Center has resources.
For tech help you can chat with xxx or xxxx to see if either of them can help
and/or you can call the HUIT help
desk.