Discover if economics is a subject for you...

School is out, does it hurt us?

Blog > Published in COVID-19, April 30th, 2020

by Simon Burgess, University of Bristol, and Hans Sievertsen, University of Bristol & VIVE

1. Interrupted teaching

You are probably reading this at home and not at school or in a library, because schools and universities have been closed around the world because of COVID-19. Teaching has been interrupted, cancelled or moved online, but will this have long-term consequences for the affected students?

Can a few weeks (or maybe months) of teaching make a difference?

Is it really harmful if school is out for a short period of time? Does it really affect the skills of students? Many politicians, parents and students are asking themselves that question right now.

To answer this question, we could compare outcomes of two individuals, Alice and Bob, where Alice attended a few more weeks of schooling. By “outcomes” we mean measures such as life satisfaction, earnings, unemployment or sick days. It is not unlikely that Alice would have better outcomes in comparison to Bob. However, such a comparison would not necessarily be informative about whether the few extra weeks of teaching caused the better outcomes. There could be a reason for why Bob attended fewer weeks of schooling and the very same reason could also be the explanation for worse health outcomes. Such reasons are called “confounding variables”, because it confounds the “causal relationship” we are interested in.

At the end, the data would show that Alice (who attended more schooling) has better outcomes, but because we didn’t account for the factors that made her attend more schooling than Bob, it could be the case that Alice would have better outcomes even if they didn’t attend more schooling. A simple comparison between Alice and Bob is in that case therefore not useful to get insights about whether more schooling causes better outcomes.

Now you might think: Wait a minute, isn’t the problem that we are only comparing Alice and Bob and that we should add a lot more individuals (what we call observations) to our analysis? It is certainly right that we wouldn’t be able to make any conclusions based on just Alice and Bob. But even if we also added Carol and David and a million other people, we wouldn’t solve the problem above. Adding more observations is important to sort out whether the better outcomes for Alice was just a coincidence or whether this is a true pattern.  More data enables us to say whether there was a correlation between schooling and life satisfaction, but not whether there is a causal relationship between schooling and life satisfaction. If Bob types are systematically different to Alice types; for example because attending less school is correlated with a confounding variable such as health or resources at home, the correlation between schooling and outcomes can be very different to the causal relationship between these two variables.

Question: What could be a reason for both: (A) Bob attending fewer lectures than Alice & (B) Bob having lower earnings, lower life satisfaction and more sick days in comparison to Alice? (This is what we call a confounder, or a variable that leads to an “omitted variable bias”)

The quest for the causal effect of schooling: The key problem in the comparison between Alice and Bob is that we don’t know why Alice stayed in school for longer. But what if we knew the reason and we knew that this reason could not affect their later life wellbeing?

A couple of years ago, a group of Swedish economists1 found such a reason: They wanted to know whether a couple of days in school make a difference for how well you do on a test. To answer this question they look at variation in the number of school days you have completed at the time of a very specific test: in preparation for the Swedish Military Service, all 18-year old men attended a test. The exact timing of the test varied, which led to variation in the number of school days (See Figure 1) and the researchers knew that after controlling for a few variables (for example date of birth), whether a person had 200 days of schooling more or less, was as good as random. We say that the variation in school days was “conditionally random” (conditionally means after we’ve accounted for the key variables).

Figure 1 from Carlsson et al (2015): “The effect of schooling on cognitive skills” (see footnote 1).

Looking at Figure 1 again, we can imagine that Alice had 200 school days and Bob 150 and then look how they did on the test. Unsurprisingly the researchers found that Alice did quite a bit better than Bob in the test. This suggests that they actually learn something in a school.

Question: The Swedish economists (see footnote 1) used variation in school days that was caused by variation in the timing of a test. The difference in school days between Alice and Bob was in this case therefore not due to underlying differences between Alice and Bob, and we can attribute difference in outcomes to the timing of the test. Could you find another reason for why some people attend more education than others? This reason should not be linked to individual differences between individuals (such as their interest in studying, their parents’ income or their study skills).

Experimenting with online teaching

Box 1: Methods for Finding causal effects

Economists (and social scientists in general) are often interested in whether a specific policy (for example schooling) has an effect on some outcome (for example earnings or life satisfaction. So how can we sort out whether a policy really affected an outcome? Two of the most popular approaches are:

1) Randomised control trials where researchers randomly (for example by rolling a dice) allocate individuals to treatments. This works because the random assignment breaks the systematic link between treatment participation and other factors.

2) Quasi-experimental approaches
Instead of rolling dice to randomly allocate students into treatment, we sometimes have institutional features that randomly allocate individuals to treatment. We can then use regression techniques such as instrumental variables or difference-in-differences to find the effect of the treatment.

Many schools and universities are replacing at least parts of normal teaching with online teaching. But is online teaching as good as normal teaching?

Again, we could imagine comparing the exam performance of Alice and Bob, where Alice attended an online course and a Bob the same course taught in-person. But again, just like in the example above, this will only tell us something about whether delivery mode (online or in person) is correlated with exam performance; it would not tell us anything about whether a worse or better exam performance was caused by the delivery mode. It might be the case that there are reasons for why Alice and Bob attended different teaching formats, and some of the very same reasons can explain differences in how well they do on the exam. We could look for some “quasi-experimental” variation like the Swedish researchers did. We call it quasi-experimental, because it wasn’t really an experiment, but it was close.

In the case of online teaching, a couple American researchers actually carried out a real experiment.2 In a so called “Randomized control Trial (RCT)”, the researchers assigned students in a microeconomics course to three types of teaching by a lottery. Alice was in an online microeconomics course because she drew a lottery number assigning her to that, Bob was in a normal course, because he drew a lottery number that assigned him to that type of course. Unfortunately, their results show that learning outcomes were reduced for the students who attended the online courses compared to the normal courses. However, your teachers probably are working intensively to avoid this happening to you!

Question: Randomized control trials, where researchers randomly assign individuals to a specific treatment (such as online teaching) are often considered as the best method to get a measure of the effect of such a treatment. However, in many cases researchers rely on other methods (like in the Swedish case). Why do you think that is the case? What could be reasons for not carrying out more experiments?

2. Interrupted exams 

Box 2: Economic concepts: statistical discrimination

Statistical discrimination occurs when we judge an individual based on an average characteristic of a group the individual is related to.

A famous example of statistical discrimination is the “ban the box” policy in the United States, where employers are not allowed to ask about a job applicant’s criminal records. However, evidence shows that the policy had harmful effects. What can employers do, if they aren’t allowed to ask for individual crime records? They can check which demographic groups of the population that has most criminal records and don’t invite them for a job interview.

Indeed, a study has showed that “ban the box” policies reduced the employment probability of you low skilled black men by 5.1 percent.

Statistical discrimination is often a rational strategy, but it can be very harmful for the individual and lead to substantial inequalities between groups.

Doleac, Jennifer L. and Benjamin Hansen. “The unintended consequences of “ban the box”: Statistical discrimination and employment outcomes when criminal histories are hidden.” Journal of Labor Economics 38.2 (2020): 321–374

Statistical discrimination in assessments.

The school closures also coincide with a key assessment period and many exams have been postponed or cancelled.  The UK has, for example, cancelled the exams for the main public qualifications, GCSEs and A levels, for the entire cohort. Depending on the duration of the lockdown, we will likely observe similar cancellations around the world. One potential substitute for the cancelled assessments is to use teacher assessments. However, evidence from various settings show that this solution might be harmful for some individuals. Specifically, evidence suggests that teachers tend to assess students more generously if the student belongs to a group that usually does well in this subject.3 For example, if girls do well in English on average, girls will be assessed more generously in that subject compared to boys. This is harmful for the boys who actually are really good in English.

Do teachers do this because they don’t like boys? No! In most cases teachers hopefully have enough information about the individual student to give an accurate mark. But in some cases, the teachers have limited information available on the individual, and they (subconsciously) rely on the signals they have available (whether the student comes from a group that usually does well). This is what is called statistical discrimination. Most of us use statistical discrimination in our everyday life. We decide not to take the car on Friday afternoon, because usually it is very crowded with cars on a Friday afternoon. We don’t check the actual traffic level, but simply use the weekday to get a (good) guess of the traffic levels. However, in the case of assessments we should avoid statistical discrimination and instead give each student a fair and individual assessment.

Question: Can you think of other examples of harmful uses of statistical discrimination in society?

The signalling value of education 

Some people might actually benefit from the COVID-19 interruptions. For example, in Norway, it has been decided that all 10th grade students will be awarded a high-school degree. And research4 shows that the 1968 abandoning of the normal examination procedures in France (following the student riots) led to positive long-term labour market consequences for the affected cohort.

Why is it not unreasonable to think that such a policy actually leads some individuals to acquire more skills? Well, one reason for going to university is to show that you are able to attend and complete university for future employers (and partners?). Completing university is a “signal that you are smart”. Now, imagine what happens in Norway if you attended high school only to show that you are smart enough to graduate, but because of COVID-19 everyone completes high school. You can’t use your high school diploma as a signal to stand out anymore, because everyone passed. So, what do you do to signal you are smart? You attend the next level (college or university) and complete it. By letting everyone pass high school, Norway might increase the number of students that enrol in university for this cohort. This is simply because education is used as a signal of how smart you are.

Box 3: Human capital vs. signalling

A big question in economics is whether schooling and education causes people to earn more money because it makes them better workers (the human capital theory) or simply because it allows them to signal that they are smart (the job-market-signalling theory).

Since the 1960s and when researchers came up with these theories about the link between education and earnings, a lot of research has investigated whether the human capital or signalling model is true. Today most economists agree that both reasons are likely to be true. Yes, employers use signals such as educational credentials to decide whom they want to hire (note this is just another example of statistical discrimination). But employers tend to learn about the workers’ skills quite fast and what they learned in school matters. So just keep on studying!

The cancellation of exams can also affect the signalling value of a degree if the online exams have greater uncertainty than normal exams. An experiment5 (just like the experiment with online teaching) shows that employers use grade point averages when they decide which job applicant to invite for a job interview.

Importantly if the exam mark is inaccurate because of the move to online exams or alternative solutions, it hurts not only the individual, but also society in general. The individual might not get the mark that they deserve for their performance, and they might therefore get the right job. Employers might not hire the right worker, because the grade point average (the average of the applicants’ marks) contained noise. We say that the “matching” between employers and workers is poor. Because employers and workers are not matched as well, meaning that workers have to spend more time finding the right job and employers more time to find the right worker. This is inefficient for everyone.

Question: The signalling theory is not only used by economists, but also in biology and evolutionary sciences. Just think about the role of the bird’s singing in the mating process. Why do you think a bird that is able is to sing nicely is an attractive mate?

Links from the Financial Times

You may be able to get free access to these links via your school.

Footnotes

1. Carlsson, M., Dahl, G. B., Öckert, B. and Rooth, D. (2015) “The Effect of Schooling on Cognitive Skills” Review of Economics and Statistics vol. 97(3) pp. 533–547

2. Alpert, William T., Kenneth A. Couch, and Oskar R. Harmon. “A randomized assessment of online learning.” American Economic Review 106.5 (2016): 378-82.

3 – Burgess, Simon and Greaves, Ellen, (2013), Test Scores, Subjective Assessment, and Stereotyping of Ethnic Minorities, Journal of Labor Economics, 31, issue 3, p. 535 – 576
– Rangvid, Beatrice Schindler. “Systematic differences across evaluation schemes and educational choice.” Economics of Education Review 48 (2015): 41-55.

4. Maurin, Eric, and Sandra McNally. “Vive la revolution! Long-term educational returns of 1968 to the angry students.” Journal of Labor Economics 26.1 (2008): 1-33.

5. Piopiunik, Marc, et al. “Skills, signals, and employability: An experimental investigation.” European Economic Review 123 (2020): 103374.

«
»