316-657 Special Topics in Advanced Econometrics

POLICY AND PROGRAMME EVALUATION AND DESIGN
Semester 2, 2005
COURSE OUTLINE

Lecturers

Jeff Borland

Room 505, Economics and Commerce Building
Telephone: 8344 5300
Email: [email protected]

Yi-Ping Tseng

Room 618, Alan Gilbert Building
Telephone: 8344 2132
Email: [email protected]

Hielke Buddelmeyer

Room 609, Alan Gilbert Building
Telephone: 8344 2091
Email: [email protected]

Roger Wilkins

Room 610, Alan Gilbert Building
Telephone: 8344 2092
Email: [email protected]

Subject web page

http://www.economics.unimelb.edu.au/subject_pages/2005/semester2/316-657/316-657.htm

Lectures

Tuesdays and Thursdays 5.15-6.45 (Lower Theatre, Land and Food Resources)

Course description

This course provides an introduction to methods for the evaluation of (government) policies and programmes. While conceptual and methodological issues will be covered, the course will have an applied focus, examining existing empirical evaluations and providing practical guidance on implementation. Particular emphasis will be given to the evaluation of labour market policy interventions.

The methods covered in the course comprise experimental and quasi-experimental methods (both observational approaches) and behavioural microsimulation modelling (ex ante evaluation). Experimental and quasi-experimental methods represent a variety of approaches for ex post estimation of the impact of a programme or policy, all of which seek to mimic' a controlled experiment. The behavioural microsimulation methods examined in the course incorporate behavioural responses, especially labour supply responses, into the analysis of the effects of the tax and transfer system. These methods involve estimation of labour supply and wage models, which can then be used to predict changes in labour supply of different groups in the community in response to proposed tax and/or transfer policy changes.

Reading

There is no required text book. References are listed in the detailed course outline which follows. Only references marked with an asterisk should be considered essential reading, although lecturers may add to the list of required reading during semester.

Assessment

Exam (50%): This will be held during the examination period and will cover all material covered in the course.

3 problem sets (30%) : A problem set will be assigned by each of the last three lecturers on their respective sections of the course.

Review of a journal article (20%): A short report (approximately five pages) which critically reviews an article or working paper. A list of suggested articles will be provided, but students may choose another article that is approved by one of the lecturers in advance. Students will also be required to give a short (15 minute) presentation of the report in the last week of semester. The report is due 25 th October. Assessment will be based on both the presentation (5%) and the report (15%).

Overview of topics covered

Detailed course outline (by week) and reading list

General Reading

Holland , P. (1986) Statistics and Causal Inference, Journal of American Statistical Association , 81, pp.945-970. (including comments and rejoinder)

Heckman, J., LaLonde, R. and Smith, J. (1999), The Economics and Econometrics of Active Labour Market Programs, in A. Ashenfelter and D. Card, eds., Handbook of Labour Economics, vol. 3, Amsterdam : Elsevier Science.

Angrist, J. and Krueger, A. (1999) Empirical Strategies in Labor Economics, in A. Ashenfelter and D. Card, eds., Handbook of Labour Economics, vol. 3, Amsterdam : Elsevier Science.

Blundell, R. and Costas Dias, M. (2000) Evaluation Methods for Non-Experimental Data, Fiscal Studies , 21(4), 427-68.

Cobb-Clark, D. and Crossely, T. (2003) Econometrics for Evaluations: An Introduction to Recent Developments, The Economic Record , 79(4), 491-511.

Week 1 (Jeff Borland)

Introduction

Overview

The evaluation problem

Parameters of interest

Social experiments

Reading

*Borland, J., Tseng, Y. and Wilkins, R. (2005) Experimental and Quasi-Experimental Methods of Microeconomic Program and Policy Evaluation, Melbourne Institute Working Paper 08/2005.

Week 2 (Yi-Ping Tseng)

Social experiments vs. regression vs. matching

Social experiments

Non-experimental evaluation: cross-sectional comparisons

Regression

Identification assumption

Regression models for different types of outcome variables

Matching

Identification assumptions

Character matching (exact matching)

Mahalanobis matching

Propensity score matching

Reading

*Wooldridge, J. (2000) Introductory Econometrics: A modern Approach, South-Western College Publishing. Chapter 3 (pp. 66-81). Chapter 17 (pp. 529-546)

*Heckman, J. and J. Smith (1995), Assessing the Case for Social Experiments. Journal of Economic Perspectives , 9(2), pp.85-100.

*Breunig, R., Cobb-Clark, D., Dunlop, Y. and M. Terrill (2003), Assisting the long-term unemployed: Results from a randomized trial', Economic Record , 79, 84-102.

*Rubin, D. (1980), Bias Reduction Using Mahalanobis-Metric Matching , Biometrics , 36(2), pp. 293-298

*Rosenbaum, P. and D. Rubin (1983) The Central Role of the Propensity Score in Observational Studies for Causal Effects', Biometrika , 70, 41-55.

Week 3 (Yi-Ping Tseng)

Propensity score matching

Estimation of propensity score and specification tests (balancing tests)

Common support problem

Different matching algorithms

Matching quality (after matching balancing test)

Standard errors

Multiple treatment matching

Reading

*Lechner, M. (2001) A Note on the Common Support Problem in Applied Evaluation Studies University of St. Gallen Discussion Paper 2001-01.

*Heckman, J., H. Ichimura, and P. Todd (1997) Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme', Review of Economic Studies 64(4), 605-54.

Heckman, J., H. Ichimura, and P. Todd (1998a), Matching as an Econometric Evaluation Estimator', Review of Economic Studies 65(2), 261-94.

*Rubin, D. and T. Neal (2000), Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates, Journal of the American Statistical Association , 95(450), pp.573-585.

*Dehejia, R. and S. Wahba (1999) Causal effects in Nonexperimental studies: Reevaluating the Evaluation of Training Programs', Journal of the American Statistical Association, 94, 1053-1062.

*Smith, J and P. Todd (2005), Does matching overcome Lalonde's critique of nonexperimental estimators?', Journal of Econometrics 125(1-2), pp.305-353 . (including comments and rejoinder)

Lechner, Michael (2000), An Evaluation of Public-Sector-Sponsored Continuous Vocational Training Programs in East Germany , Journal of Human Resources, 32(2), pp. 347-375.

*Lechner, M., R. Miquel and C. Wunsch (2005), Long-run effects of public sector sponsored training in West Germany ', IAB discussion paper No. 3, 2005.

Week 4 (Yi-Ping Tseng)

Practical issues in implementing matching methods

Examples

Software for matching

Programming using Stata (computer lab)

Exercises

Reading

*Smith, J. (2000), A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies', Swiss Journal of Economics and Statistics . 136(3):1-22.

*Black, D. and J. Smith (2004), How robust is the evidence on the effects of college quality? Evidence from matching, Journal of Econometrics , 121, pp. 99-124.

* Heckman, J. and J. Smith (1999), The Pre-programme Earnings Dip and the Determinants of Participation in Social Programme. Implications for Simple Programme Evaluation Strategies, The Economic Journal ,109, pp. 313-348.

*Sianesi, B. (2001), An Evaluation of the Active Labour Market Programmes in Sweden ', The Review of Economics and Statistics , 86(1), 133-155.

*Borland, J. and Y. Tseng (2004), Does 'Work for the Dole' Work?, Melbourne Institute Working Paper No. 14/2004, 40 pp.

Borland, J. and Y. Tseng (2003), How Do Administrative Arrangements Affect Exit from Unemployment Payments? The Case of the Job Seeker Diary in Australia , Melbourne Institute Working Paper, No. 27/2003, 57 pp.

Week 5 (Yi-Ping Tseng)

Before-after comparisons and difference-in-differences (DID)

Before and after comparisons

Identification assumptions

Implementation (regression vs. matching)

Difference-in-differences (DID)

Identification assumptions

Implementation (regression vs. matching)

Reading

*Heckman, J., H. Ichimura, J. Smith and P. Todd (1998b) Characterizing Selection Bias Using Experimental data', Econometrica , 66(5), 1017-98.

Bertrand, M., E. Duflo and S. Mullainathan (2004), How Much Should We Trust Differences-in-Differences Estimates?, Quarterly Journal of Economics, 119 (1), pp. 249-275.

*Blundell, R., M. Costa Dias, C. Meghir, J. Van Reenen (2004), Evaluating the Employment Impact of a Mandatory Job Search Program, Journal of European Economic Association , 2(4), pp.569-606.

*Doiron, D. (2004) Welfare Reform and the Labour Supply of Lone Parents in Australia : A Natural Experiment Approach, Economic Record , 80(249), 157-176.

*Smith, J and P. Todd (2005), Does matching overcome Lalonde's critique of nonexperimental estimators?', Journal of Econometrics 125(1-2), pp.305-353 . (including comments and rejoinder)

Leigh, A. (2003), Employment effects of minimum wages: Evidence from a quasi-experiment', Australian Economic Review, 36, pp.361-73.

Week 6 (Hielke Buddelmeyer)

Regression Discontinuity Design

Introduction

Identification

Applications

Exercise

Reading (Weeks 6 and 7 combined)

*Angrist, J. D. and V. Lavy (1999) Using Maimonides Rule to Estimate the Effect of Class Size on Scholastic Achievement The Quarterly Journal of Economics , 114(2):535-75

Battistin, E. and Rettore, E. (2002), Testing for programme effects in a regression discontinuity design with imperfect compliance , Journal of the Royal Statistical Society A , 165(1): 1-19

*Battistin, E. and Rettore, E. (2003), Another look at the regression discontinuity design, CEMMAP Working Papers CWP01/03

Chen, M. K. and J. M. Shapiro (2004) Does Prison Harden Inmates? A Discontinuity-based Approach, Cowles Foundation Discussion Paper No. 1450 .

DiNardo, J. and D. S. Lee (2004) Economic impacts of unionization on private sector employers: 1984-2001 The Quarterly Journal of Economics , 119 (4): 1383-1442

*Hahn, J., P. Todd and W. Van der Klaauw Identification and Estimation of Treatment Effects with a Regression Discontinuity Design, Econometrica 69(1):201-209.

*Imbens, G. W. and J. D. Angrist (1994) " Identification and Estimation of Local Average Treatment Effects ," Econometrica , vol. 62(2), pages 467-75

Jacob, B. A. and L. Lefgren (2002) The Impact of Teacher Training on Student Achievement: Quasi-Experimental Evidence from School Reform Efforts in Chicago , NBER Working Paper No. 8916.

Jacob, B. A. and L. Lefgren (2002) Remedial Education and Student Achievement: A Regression-Discontinuity Analysis, NBER Working Paper No. 8918.

Lavy, V. (2002) Evaluating the Effect of Teachers' Group Performance Incentives on Pupil Achievement, The Journal of Political Economy 110(6):1286-1317.

Lee, D. S. (2004) Randomized Experiments from Non-random Selection in U.S. House Elections , forthcoming in Journal of Econometrics . Previous version: The Electoral Advantage to Incumbency and Voters' Valuation of Politicians' Experience: A Regression Discontinuity Analysis of Elections to the U.S. House , NBER Working Paper #8441

Lee, D. S. and D. Card (2004), Regression Discontinuity Inference with Specification Error , Center for Labor Economics Working Paper #74, June 2004.

Leuven , E. and H. Oosterbeek (2004) Evaluating the Effect of Tax Deductions on Training Journal of Labor Economics , 22(2):461-88

McCrary, J. and H. Royer (2005) The Effect of Maternal Education on Fertility and Infant Health: Evidence From School Entry Policies Using Exact Date of Birth, mimeo, University of Michigan .

Palangkaraya, A. and J. Yong (2004) How Effective is Lifetime Health Cover' in Raising Private Health Insurance Coverage in Australia ? An Assessment Using Regression Discontinuity, Melbourne Institute Working Paper No. 33/04.

Van der Klaauw, W. (2002) Estimating the Effect of Financial Aid Offers on College

Enrollment: A Regression-Discontinuity Approach. International Economic

Review , 43 (4): 1249-87.

Week 7 (Hielke Buddelmeyer)

Regression Discontinuity Design (continued)

Applications

Microsimulation modelling

Introduction

Underlying Discrete Labour Supply Model

Reading

RDD: As per week 6.

Microsimulation: As per week 8.

Week 8 (Hielke Buddelmeyer)

Microsimulation modelling (continued)

Applications

Exercise: Be the honourable Peter Costello MP for a day and devise a new budget. Then use a dynamic microsimulation model (provided) to calculate the costs, the winners and losers and the labour supply effects of your budget.

Reading (Weeks 7 and 8 combined)

*Creedy, J. and G. Kalb (2005) Behavioural Microsimulation Modelling for Tax Policy Analysis in Australia : Experiences and Prospects, Melbourne Institute Working Paper No 2/05.

*Labeaga, J. M., X. Oliver and A. Spadaro (2005) Discrete Choice Models of Labour Supply, Behavioural Microsimulation and the Spanish Tax Reforms, Paris-Jourdan Sciences Economiques Working Paper 2005-13.

*Van Soest, A. (1995). Structural models of family labor supply; a discrete choice approach, Journal of Human Resources , 30(1), 63-88.

Bourguignon, F. and A. Spadaro (2005) Microsimulation as a Tool for Evaluating Redistribution Policies, Paris-Jourdan Sciences Economiques Working Paper 2005-02.

Klevmarker, N.A. (2002) Micro Simulation A Tool for Economic Analysis mimeo, Uppsala University .

Lambert, S., R. Percival, D. Schofield and S. Paul (1994) An Introduction to STINMOD: A Static Microsimulation Model, STINMOD Technical Paper 1, October 1994.

Vagliasindi, P. A., M. Romanelli and C. Bianchi (Undated) Validation and Policy Analysis using Dynamic Microsimulation Modelling: Evidence from Experiments in the Italian Case, mimeo, University of Parma .

Van Soest, A., M. Das and X. Gong (2002). A structural labour supply model with flexible preferences, Journal of Econometrics , 107, 345-374.

Week 9 (Roger Wilkins)

Instrumental variables I

Basics (with binary IVs)

Common effect IV

Weak instruments

Reading (Weeks 9 and 10 combined)

*Wooldridge, J. (2000) Introductory Econometrics , South-Western College Publishing, Chapter 15.

*Heckman, J., LaLonde, R. and Smith, J. (1999), The Economics and Econometrics of Active Labour Market Programs, in A. Ashenfelter and D. Card, eds., Handbook of Labour Economics, vol. 3, Amsterdam : Elsevier Science, Section 7.

*Angrist, J. and Krueger, A. (1999) Empirical Strategies in Labor Economics, in A. Ashenfelter and D. Card, eds., Handbook of Labour Economics, vol. 3, Amsterdam : Elsevier Science. [IV section]

*Angrist, Joshua, Guido Imbens and Donald Rubin. 1996. Identification of Causal Effects Using Instrumental Variables, Journal of the American Statistical Association . 91(434). 444-472. [with discussion]

Angrist, J. and Krueger, A. (2001) Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments, Journal of Economic Perspectives , 15(4), 69-85.

Cobb-Clark, D. and Crossley, T. (2003) Econometrics for Evaluations: An Introduction to Recent Developments, The Economic Record , 79(4), 491-511 [section on IV]

Heckman, J. (1997) Instrumental Variables: A Study of Implicit Behavioural Assumptions Used in Making Program Evaluations, Journal of Human Resources , 32(3), 441-52.

Heckman, James and Edward Vytlacil (2001) Policy-Relevant Treatment Effects. American Economic Review . 91(2). 107-111.

Imbens, G. and Angrist, J. (1994) Identification and Estimation of Local Average Treatment Effects, Econometrica , 62(2), 467-75.

Angrist, J. (2004) Treatment Effect Heterogeneity in Theory and Practice, Economic Journal , 114 (March), C53-C83.

Angrist, J. and Imbens, G. (1995) Two-Stage Least Squares Estimation of Average Causal Effects in Models with Variable Treatment Intensity, Journal of the American Statistical Association , 90(430), 431-42.

Angrist, J. and Krueger, A. (1991) Does Compulsory School Attendance Affect Schooling and Earnings?, Quarterly Journal of Economics , 106(4), 979-1014.

Angrist, J. (1990) Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, American Economic Review , 80(3), 313-36.

Kling, J. (2001) Interpreting Instrumental Variables Estimates of the Returns to

Schooling, Journal of Business and Economic Statistics 19(3): 358-364.

Heckman, James J. 1979. Sample Selection Bias as a Specification Error. Econometrica , 47(1): 153-161.

Week 10 (Roger Wilkins)

Instrumental variables II

Heterogeneous effects IV (local average treatment effects)

Models with variable treatment intensity

Relationship between IV and sample selection models

Examples

Reading

As per Week 9.

Week 11 (Roger Wilkins)

Comparison of methodologies: Choosing the right approach

Some recent developments in program evaluation methods

Dynamic evaluation

General equilibrium evaluation

Reading

*Heckman, J., LaLonde, R. and Smith, J. (1999), The Economics and Econometrics of Active Labour Market Programs, in A. Ashenfelter and D. Card, eds., Handbook of Labor Economics, vol. 3, Amsterdam : Elsevier Science, Sections 8.3 and 8.4.

*LaLonde, Robert (1986) Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review . 76(4): 604-620.

*Heckman, James and V. Joseph Hotz (1989) Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training. Journal of the American Statistical Association . 84(408): 862-874 (includes comments by Holland and Moffitt).

Dehejia, Rajeev and Sadek Wahba (1999) Causal Effects in Non-experimental Studies: Reevaluating the Evaluation of Training Programs. Journal of the American Statistical Association. 94: 1053-1062.

Smith, Jeffrey and Petra Todd (2002) Does Matching Overcome LaLonde's Critique of Nonexperimental Estimators? Journal of Econometrics .

Lechner, M. (2004) Sequential Matching Estimation of Dynamic Causal Models, IZA Discussion Paper , No. 1042.

Abbring, J. and Van Den Berg, G. (2003) The Nonparametric Identification of Treatment Effects in Duration Models, Econometrica , 71(5), 1491-1517.

Fredriksson, P. and Johansson, R. (2004) Dynamic Treatment Assignment The Consequences for Evaluations Using Observational Data, IZA Discussion Paper , No. 1062.

Lise, Jeremy, Shannon Seitz and Jeffrey Smith. 2005. Equilibrium Policy Experiments

and the Evaluation of Social Programs. Unpublished manuscript.

Week 12 (Hielke Buddelmeyer and Roger Wilkins)

Student presentations, problem sets review and course review

Additional references

(Note: incomplete)

Ashenfelter, O. (1978), Estimating the Effect of Training Programs on Earnings, Review of Economics and Statistics , 60, 47-57.

Ashenfelter, O. and Card, D. (1985), Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs, Review of Economics and Statistics , 67, 648-660.

Borland, J. and Y. Tseng (2003), How Do Administrative Arrangements Affect Exit from Unemployment Payments? The Case of the Job Seeker Diary in Australia , Melbourne Institute Working Paper, No. 27/2003, 57 pp.

Borland, J. and Y. Tseng (2004), Does 'Work for the Dole' Work?, Melbourne Institute Working Paper No. 14/2004, 40 pp.

Bertrand, M., E. Duflo and S. Mullainathan (2004), How Much Should We Trust Differences-in-Differences Estimates?, Quarterly Journal of Economics, 119 (1), pp. 249-275.

Black, D., J. Smith, M. Berger and B. Noel ( 2003), Is the Threat of Reemployment Services More Effective than the Services Themselves? Evidence from Random Assignment in the UI System. American Economic Review, 93(4): pp.1313-1327 .

Black, D. and J. Smith(2004), How robust is the evidence on the effects of college quality? Evidence from matching, Journal of Econometrics , 121, pp. 99-124.

Blundell, R., M. Costa Dias, C. Meghir, J. Van Reenen (2004), Evaluating the Employment Impact of a Mandatory Job Search Program, Journal of European Economic Association , 2(4), pp.569-606.

Breunig, R., Cobb-Clark, D., Dunlop, Y. and M. Terrill (2003), Assisting the long-term unemployed: Results from a randomized trial', Economic Record , 79, 84-102.

Dehejia, R. and S. Wahba (1999) Causal effects in Nonexperimental studies: Reevaluating the Evaluation of Training Programs', Journal of the American Statistical Association, 94, 1053-1062.

Doiron, D. (2004) Welfare Reform and the Labour Supply of Lone Parents in Australia : A Natural Experiment Approach, Economic Record , 80(249), 157-176.

Heckman, J. , N. Hohmann, N. , J. Smith and M. Khoo(2000), Substitution and Drop Out Bias in Social Experiments: A Study of an Influential Social Experiment. Quarterly Journal of Economics , 115(2), pp.651-694.

Heckman, J., R. Lalonde and J. Smith (1999), The economics and econometrics of active labor market programs', pages 1865-2097 in O. Ashenfelter and D. Card (eds.) Handbook of Labor Economics Volume 3A (Amsterdam, Elsevier).

Heckman, J., H. Ichimura, and P. Todd (1997) Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme', Review of Economic Studies 64(4), 605-54.

Heckman, J., H. Ichimura, and P. Todd (1998a), Matching as an Econometric Evaluation Estimator', Review of Economic Studies 65(2), 261-94.

Heckman, J., H. Ichimura, J. Smith and P. Todd (1998b) Characterizing Selection Bias Using Experimental data', Econometrica , 66(5), 1017-98.

Heckman, J. and Hotz, V.J. (1989), Choosing among Alternatives Nonexperimental Methods for Estimating the Impact of Social programs, Journal of the American Statistical Association, 84, 862-874.

Heckman, J., Smith, J. and N. Clements, (1997), Making the Most out of Program Evaluations and Social Experiments: Accounting for Heterogeneity in program Impacts, Review of Economic Studies, 64, 487-536.

Hirano, K., G. Imbens, and G. Ridder (2003) `Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score', Econometrica , 71, 1161-1189.

Imbens, G. (2004) ` Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review, The Review of Economics and Statistics , 86(1), 4-29.

Lechner, Michael (2002), Program Heterogeneity and Propensity Score Matching: An Application to the Evaluation of Active Labor Market Policies, Review of Economics and Statistics, 84 (2), 205-220.

Lechner, M., R. Miquel and C. Wunsch (2005), Long-run effects of public sector sponsored training in West Germany ', IAB discussion paper No. 3, 2005.

Rubin, D. (1980), Bias Reduction Using Mahalanobis-Metric Matching , Biometrics , 36(2), pp. 293-298

Rubin, D. (1979), Using multivariate matched sampling and regression adjustment to control bias in observational studies', Journal of the American Statistical Association , 7, pp. 34-58.

Rosenbaum, P. and D. Rubin (1983) The Central Role of the Propensity Score in Observational Studies for Causal Effects', Biometrika , 70, 41-55.

Rosenbaum, P. and Rubin, D.B. (1984), Reducing Bias in Observational Studies Using Subclassification on the Propensity Score, Journal of the American Statistical Association, 79, 516-524.

Rubin, D. and T. Neal (2000), Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates, Journal of the American Statistical Association , 95(450), pp.573-585.

Sianesi, B. (2001), An Evaluation of the Active Labour Market Programmes in Sweden ', The Review of Economics and Statistics , 86(1), 133-155.

Smith, J. (2000), A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies', Swiss Journal of Economics and Statistics . 136(3):1-22.

Smith, J and P. Todd (2005), Does matching overcome Lalonde's critique of nonexperimental estimators?', Journal of Econometrics 125(1-2), pp.305-353 .

Dehejia, R. (2005), Practical Propensity score matching: a reply to Smith and Todd, Journal of Econometrics 125(1-2), pp.355-364 .

Smith, J and P. Todd (2005), Rejoinder', Journal of Econometrics 125(1-2), pp.365-375 .

Zhao, Z. (2004) Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence', The Review of Economics and Statistics , 86(1) 91-107.