Course Project Information
About the Project
The course project is the cornerstone of your learning experience in this module. Its key objective is to evaluate a statistical method (or a set of related methods) through Monte Carlo simulations. You are required to design, execute, and analyze simulations to assess the method’s performance in a given context.
The project is designed to help you develop four key skills:
- Designing and interpreting simulations to evaluate statistical methods.
- Organizing and sharing technical work in a reproducible and professional manner.
- Communicating findings effectively through writing and presentations.
- Staying resilient and adapting to changes in open-ended technical projects.
Your grade for the course will be based entirely on this project. For deadlines, group sign-ups, and submission details, please refer to eCampus.
Broadly, the project involves the following key steps:
- Method selection: choose a method
- Simulation design: develop and implement simulations to evaluate the method’s properties (e.g., bias, variance, coverage, efficiency, model selection abilities, etc.) across relevant scenarios.
- Analysis and interpretation: analyze simulation results and discuss their implications.
- Documentation: produce a term paper and deliver a presentation, accompanied by fully reproducible code.
Choosing a Topic
Scope
The expected statistical level aligns with modern methods covered in advanced textbooks (e.g. Chernozhukov et al. 2024; Gaillac and L’Hour 2025; Wager 2024). Projects based on recent papers are also welcome if they are feasible within the course timeframe.
You are free to choose a topic from any branch of statistics:
- Causal inference;
- Prediction;
- Inference.
The question of your project does not have to be groundbreaking, as the emphasis is more on the technical soundness. However, I would still recommend trying more “modern” things, as these will likely provide more learning value and relevance to your future work.
What Makes a Good Topic?
A strong project topic is specific and follows this pattern:
One (or more) method(s) are evaluated in one (or more) scenario(s) to answer a specific question.
A few simple examples to help explain the idea:
Evaluating the Callaway and Sant’Anna (2021) estimator.
- Question: Callaway and Sant’Anna (2021) estimator is designed to estimate average treatment effects in settings with staggered treatment timing and heterogeneous effects. How does it perform in terms of bias and variance across different sample sizes and treatment effect distributions?
- Approach: simulate data with varying sample sizes, treatment effect heterogeneity, and treatment patterns. Tabulate bias, variance, and coverage rates.
Efficiency Loss: Callaway and Sant’Anna (2021) vs. Two-Way Fixed Effects
- Question: In settings where the two-way fixed effects (TWFE) estimator is consistent, how much efficiency is lost by using the Callaway and Sant’Anna (2021) estimator instead?
- Approach: Design DGPs where TWFE is consistent (e.g. no staggered adoption). Vary sample sizes, treatment effect distributions, and error structures to quantify efficiency trade-offs.
Bias in Cross-Validation for Model Selection
- Question: Cross-validation (CV) is the de facto standard for model selection in predictive settings. However, CV is inherently biased, as \(K\)-fold CV gives an unbiased estimator for the performance of the algorithm when faced with share \((K-1)/K\) of the training data. However, the final selected algorithm gets retrained on the full dataset. How large is the bias in risk, and does model selection based on biased risk estimates lead to meaningfully worse predictive performance?
- Approach: Simulate prediction tasks with varying signal-to-noise ratios, sample sizes, and model complexities. For a given sample size \(N\), compare cross-validation-based model selection based to selection based on a new validation sample of size \(N\).
Format
Group Work
Group formation:
- You will work in groups of three students.
- Form your groups independently, but notify the instructor if you need help finding teammates.
The key component of the project phase is group meetings. These meetings will by default take place during lecture hours. Every group should meet with the instructor at least once every two weeks. The sign-up details will be provided on eCampus.
These meetings are your opportunity to:
- Discuss progress and challenges.
- Get feedback on your project direction.
- Figure our directions for improvement.
The general idea is to rapidly iterate on the project based on the feedback and discussions in the meetings.
Open-ended technical projects often evolve as you work on them. Be prepared for a continuous learning process, including:
- Refining your code and simulation design (e.g. implementation, refactoring).
- Deepening your understanding of statistical methods and their real-world context.
- Learning more technical tools as challenges arise.
- Adjusting your evaluation approach as needed.
- Shifting scope if earlier approaches prove ineffective.
Collaboration within your group is key.
You can think of this stage as an agile (learning) experience:
- Regular check-ins to assess progress.
- Rapid adjustments based on feedback.
- Learning just what you need for the next step.
- Collaborating on both technical work and knowledge sharing.
The aim is to produce a final project that reflects your learning and iteration throughout the course.
Term Paper
- Length: Maximum 20 pages (excluding bibliography and appendix).
- Note: 20 pages is an upper limit, not a target. Focus on clarity and conciseness.
- Formatting: Use 12pt font, 1.5-line spacing, and 2.5 cm margins.
- Appendix: Use for long tables, additional figures, or supplementary material.
Replication Code
- Submit your code as a GitHub repository (private or public). Add the instructor as a collaborator using their institutional email, if using a private repo.
- Include all files necessary for full replication, along with a README with clear instructions.
Evaluation Criteria
Your final grade will be based on two components:
1. Term Paper and Replication Code (70%)
| Criteria | Weight | Details |
|---|---|---|
| Method Description | 15% | Clearly describe the method, its context, and key contributions from the literature. |
| Writing Quality | 25% | Write in a clear and structured manner. State explicitly what question your simulation project is answering. |
| Code Organization and Replicability | 25% | Write modular, well-documented, and easy to replicate code. |
| Simulation Design | 20% | Design data-generating processes that are informative and relevant to the question. |
| Analysis and Interpretation | 15% | Rigorously interpret simulation results. Discuss limitations and implications. |
2. Presentation (30%)
| Criteria | Weight | Details |
|---|---|---|
| Content Understanding | 60% | Demonstrate a comprehensive understanding of the method and simulation design. |
| Clarity and Delivery | 40% | Present in an organized, accessible manner. Use legible slides. |