10/30/2016 POLICY NOTE A longitudinal analysis linking test scores and later cognitive and socio-emotional outcomes among Cambodian youth Key questions In this note, we examine how Cambodian youth’s test score performance during lower secondary relates to their success several years later as they transition from schooling to work. Specifically, we explore the association between cognitive skills measured using math and vocabulary test scores during 8th grade and several measures of “success” including test scores and socio-emotional well-being three years later. We explore the following questions: 1. How persistent are test scores? Controlling for individual, household and school characteristics, how well do test scores in 8th grade predict test scores 3 years later? 2. Are test scores measured in 8th grade associated with more years of schooling? 3. How do test scores relate to later schooling versus work decisions? 4. Are test scores measured in 8th grade associated with mental health and subjective social status 3 years later? Dataset We take advantage of an existing two-wave panel dataset of students in Cambodia. This data originates from two evaluations done by Filmer and Schady in their 2011 and 2014 papers (Filmer and Schady 2011; Filmer and Schady 2014). These papers assessed the effect of a cash-transfer scholarship program by comparing recipients versus non-recipient children, where recipient status was determined by students’ positions on either side of an eligibility cut-off score. The sample of Cambodian youth in the dataset attended relatively more disadvantaged lower secondary schools throughout the country in 2005. The initial schools were chosen for the scholarship program because they served poor areas and had high levels of non-enrollment and dropout. At the start of the study, students filled out application forms which determined whether they would receive a scholarship. The first follow-up data collection occurred around 2-3 years later in 2007-08 when students were supposed to be in grade 8. This survey collected detailed information on test scores (math and vocabulary), socio-economic and demographic variables, and questions on mental health. The second follow-up occurred around 5 years later in 2010, corresponding to the latter half of grade 11, and collected information on a large number of child outcomes including years of completed schooling, employment information, and three tests (math and vocabulary and additional ravens puzzle test). The median age of children at the time of the second follow-up survey was 19. 1 10/30/2016 In this analysis, we use the two follow-up rounds in 2007-08 and 2010 to create a balanced panel of 2788 students with test score evaluations in both time periods. We use socio-economic and demographic variables collected at baseline (when the students first filled out the applications). Though the timing of the follow-up survey rounds correspond to grades 8 and 11 respectively, some initial participants may not be still enrolled in school during these subsequent surveys. As such, to use descriptors consistent with our entire sample, we will refer to the baseline, first and second follow-up as time periods 0, 1 and 2, respectively. Methods Our main independent variable of interest is test scores during time period 1, which we measure with a 14-item questionnaire for math and 128-item questionnaire for vocab. To standardize tests across time periods, for each test we calculate the percentile ranking of each individual relative to their peers. We then take the average of the two percentile rankings and use this composite measure for test scores. Our outcome variables are the following: Test score is measured the same way as time period 1 test score. At period 2, the math questionnaire consisted of 20 questions and vocabulary questionnaire consisted of 72 items. The math test covered algebra, geometry, and several questions that required using mathematical tools to answer simulated real-world situations, such as reading a simple graph or interpreting a bar chat. The vocabulary test required respondents to identify the picture corresponding to a word which the enumerator read out loud. As with midline test score, we calculate the percentile ranking of each individual relative to their cohort and take the average of the two. Mental Health is measured using the General Health Questionnaire, a widely-used and internationally validated 12-item questionnaire designed to detect current diagnosable mental and psychiatric disorders (Goldberg, 1992). It contains questions such as “Over the past month have you been feeling unhappy or depressed?”). Respondents answer “always,” “sometimes,” or “never,” which we code with values of 1, 2, or 3. The total score is the sum of all 12 scores, where a higher score corresponds to better mental health. We use this aggregate measure to again calculate the percentile ranking across individuals and use this as the outcome measure. Subjective social status is measured using two sets of questions using the MacArthur “ladder” scale of subjective social status (Adler and Stewart, 2007). Respondents are asked to place themselves on a rung of a ladder representing socio-economic status, ranging from 1 (very poor) to 10 (very rich). The first question asks them to position themselves relative to their village or neighborhood. We ask them to do this for both current perceived, as well as future expected social standing. The second scale presents the same ladder, but asks respondents to compare their socio-economic status to the rest of Cambodia. Employment and Schooling is measured both using a continuous variable for years of schooling completed and a categorical outcome variable that coded in the following way: 0 = dropped out of school and not working, 1= not in school, working but not for pay, 2= not in school, does some work for pay, 3= enrolled in school. 2 10/30/2016 We use regression analysis for all questions. We control for scholarship recipient status, and for each outcome, we run three regressions first with controls at the individual level, then with additional controls at the household/family levels, and further controls for secondary school characteristics and school indicators. Individual and household controls include age, gender, an indicator for living with mother and father, indicators for whether the mother attended school, whether the father attended school, whether the parents are civil servants, the number of children in the household, and various indicators for asset ownership. School-level controls were collected at period 1 and include student-to- teacher ratio, the share of teachers with at least upper secondary education, share of teachers with less than 5 years of service, and the ratio of classrooms to students enrolled in the school. For the mental health and subjective social status outcomes (question 4), we additionally control for period 1 mental health. For the years of schooling attained and school and work outcomes, we restrict the sample to only those who were enrolled in school at period 1, the time test scores were first collected. For question 2, we use a multinomial logit model. Since the outcome variable is coded ordinally, we explored using an ordinal logit model. However, the proportional odds assumption does not hold in our case (the null of proportional odds was rejected using the Brant test). Hence we fit a multinomial model and compare odds ratios to the base category of “neither schooling nor working”. Key Findings Cognition as measured by test score percentile rankings among Cambodian youth is persistent. Figure 1 shows the linear prediction from the regression of period 1 test score percentile on period 2 test score percentile and the full set of controls, compared with the 45-degree line of perfect association. Figure 2 shows the average period 2 percentile across 5 quintiles of period 1 scores. There is a clear positive trend: those who score higher than their peers in the first period tend to maintain their relative ranking three years later. However, the association is less than unitary: a 10 percentage point higher ranking in math and verbal scores at period 1 is associated with about a 5.9 percentage point higher ranking in period 2. Adding household and school fixed effects accounts for more of the residual error but does not significantly change the coefficient estimate (Table 1). Part of this appears to be driven by a “regression to the mean” effect, as shown in Figure 3, which plots the difference in percentile scores between periods against period 1 quintiles. Those who scored relatively poorly in period 1 see improvements whereas those who did well relative to their peers in period 1 do slightly worse in period 2. Interestingly, older youth do worse on tests than their younger peers, all else equal. A plausible explanation for this could be that age is positively correlated with grade repetition and/or dropout. Youth who do better on tests tend to stay in school for longer. A 10 percentage point increase in test score percentile ranking in period 1 is associated with about .13 more years of schooling three years later. The magnitude and significance of this association strongly persists even after adding school fixed effects, suggesting this variation is not due to observable factors that vary at the school level, such as school inputs or average socio-economic status of students in a school. 3 10/30/2016 Fewer youth who score at the highest quintile are idle and more remain in school three years later, compared to those who score at lower quintiles. Among those who are enrolled in school at period 1, 24 percent of those who score in the bottom quintile of math and verbal tests are “not in school and not working” three years later compared to 14 percent of those in the top quintile (Figure 3). Conversely, 15 percent of those in the bottom quintile are still in school three years later, compared to 48 percent of those in the top quintile. Fewer high quintile performers do unpaid work later – 26 percent of those in the bottom quintile are not in school and only doing unpaid work compared to 15 percent of those in the top quintile. Average test score performers (3rd quintile) have the highest proportion of people who are not in school but doing some paid work (38 percent). Given one percentage point higher test scores, the relative odds of being in school rather than idle are 1.024 – 1.030 times what they were before the change (Table 3). Rescaling the test score variable such that one unit corresponds to a 10 percentage point change in percentile, we find that the odds are 1.27 to 1.35 higher for being in school relative to neither schooling nor working. Higher test scores are also associated with higher odds of doing both paid and unpaid work compared to idleness, but neither of these is significant at conventional levels. Test scores in period 1 is significantly associated with mental well-being in period 2, and this association persists for some specifications even after controlling for period 1 mental health. A 10 percentage point increase in math and verbal percentile ranking relative to peers in period 1 is associated with a 0.75 to 0.92 relative increase in percentile ranking of mental health in period 2. When we control for period 1 mental health, the association reduces by more than one third but remains significant when controlling for individual and family/household variables. Significance drops to 10 percent after controlling for school characteristics and school fixed effects (Tables 4A and 4B). Youth who perform better on tests more optimistic about achieving a higher social status in the future when comparing themselves to their immediate village or neighborhood. Youth who do better on tests in period 1 do not perceive themselves to have a higher social status when asked how they currently compare to people in their village or neighborhood in period 2, but are optimistic for the future. A 10 percentage point increase in test score percentile ranking is associated with reporting .06-.07 steps higher on the 11-step ladder scale of social status (Tables 5A and 5B). The significance of the association persists when adding household and school controls, and even after including period 1 mental health, though the magnitude decreases slightly to 0.06. Interestingly, the reverse is true when youth assess their economic status relative to the nation at large: youth who perform better on tests place themselves on a lower socio-economic rung both for the present and future. When asked how they would rank themselves relative to Cambodia, higher scorers in period 1 are significantly more likely to put themselves on a lower step three years later compared to those who do worse on tests in period 1. A 10 percentage point increase in test scores is associated with .07-.08 steps lower for current perceived social status and .04-.05 steps lower for future expected social status (Tables 6A and 6B). 4 10/30/2016 There are many possible explanations for the different coefficient signs when comparisons are being made relative to a local group versus a more “global” one (the nation). It is important to recall too that our sample of youths is relatively poorer than the average in Cambodia, based on how the initial schools were selected. One possibility for these findings is that the youths who score higher on tests are more aware of the world beyond their immediate neighborhood and hence are able to make more critical and realistic comparisons. We cannot say from our data whether this awareness is generated through greater learning and hence through our independent variable of interest (test scores), or through unobserved factors; For instance, relatively higher socio-economic wealth may correlate with test scores but also correlate with owning assets like a TV or radio that increase exposure to national news (this is only an example, since in our regressions we do control for these asset ownership variables). References Adler, N. E., & Stewart J. (2007). The MacArthur scale of subjective social status. Retrieved 10/24/2016 http://www.macses.ucsf.edu/research/psychosocial/ Filmer, D. and Schady, N., 2011. Does more cash in conditional cash transfer programs always lead to larger impacts on school attendance?. Journal of Development Economics, 96(1), pp.150-157. Filmer, D. and Schady, N., 2014. The medium-term effects of scholarships in a low-income country. Journal of Human Resources, 49(3), pp.663-694. Goldberg, D., 1992. General health questionnaire (GHQ-12). Windsor, UK: Nfer-Nelson. 5 10/30/2016 Figure 1. Linear prediction of period 2 test scores Linear prediction of period 2 test scores 100 80 60 40 20 0 0 20 40 60 80 100 period 1 score (pctile) Red line is the linear regression prediction. Black line is the 45-degree line for reference. Figure 2. Test scores, average percentile in period 2 by quintile in period 1 Period 2 mean pctile on Period 1 quintiles 80 60 period 2 pctile 40 20 0 1 2 3 4 5 6 10/30/2016 Figure 3. Test scores, difference in period 2 and period 1 percentile, by period 1 quintile Diff in period 2 and 1 scores, on period 1 quintiles 15 10 Diff in scores 5 0 -5 -10 1 2 3 4 5 Figure 4. Schooling and work outcomes, by period 1 test score quintile Schooling and Work Outcomes, by Midline Test Quintile .5 0.48 .4 0.38 0.35 0.35 0.34 .3 % 0.28 0.26 0.25 0.24 0.24 0.23 0.23 .2 0.20 0.19 0.19 0.19 0.18 0.15 0.15 0.14 .1 0 1 2 3 4 5 Not in school, not working Not in school, doing unpaid work only Not in school, doing some paid work In school 7 10/30/2016 Table 1. Persistence of Test Scores From Period 1 to Period 2 Math & Verbal score (period 2 pctile) (1) (2) (3) (4) Math & Verbal score (period 1 pctile) 0.596*** 0.588*** 0.592*** 0.530*** (0.018) (0.018) (0.019) (0.029) received scholarship -0.629 0.785 0.252 -0.813 (0.850) (0.837) (0.944) (0.952) age -1.428*** -1.441*** -1.762*** -1.478*** (0.290) (0.285) (0.316) (0.321) male 3.643*** 4.935*** 4.563*** 3.987*** (0.863) (0.940) (1.099) (1.079) test score x enrolled in period 1 0.065* (0.033) enrolled in period 1 2.959 (1.797) household controls YES YES YES school controls & FE YES YES Observations 2,788 2,788 2,788 2,788 R-squared 0.404 0.413 0.437 0.449 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 8 10/30/2016 Table 2. Test Scores in Period 1 and Years of Schooling Attained (1) (2) (3) Math & Verbal score (pctile) 0.013*** 0.013*** 0.013*** (0.001) (0.001) (0.001) received scholarship 0.076 0.090 0.056 (0.055) (0.069) (0.069) age -0.111*** -0.114*** -0.151*** (0.023) (0.023) (0.021) male 0.171** 0.178** 0.118 (0.075) (0.080) (0.088) household controls YES YES school controls & FE YES Observations 1,978 1,978 1,978 R-squared 0.096 0.106 0.194 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Note: Sample is restricted to those who are enrolled in school in period 1 9 10/30/2016 Table 3. Test scores in period 1 and schooling and work outcomes in period 2 (1) (2) (3) 1 2 3 1 2 3 1 2 3 Math & Verbal score (pctile) 1.002 1.001 1.024*** 1.002 1.002 1.025*** 1.001 1.005 1.030*** (0.003) (0.003) (0.003) (0.003) (0.003) (0.003) (0.003) (0.004) (0.004) received scholarship 0.796 0.959 0.790 0.745 0.885 0.790 0.765 0.951 0.727 (0.141) (0.150) (0.125) (0.148) (0.162) (0.161) (0.165) (0.167) (0.149) age 0.987 0.950 0.769*** 0.988 0.957 0.762*** 1.000 0.913 0.671*** (0.053) (0.061) (0.056) (0.057) (0.063) (0.057) (0.068) (0.059) (0.054) male 0.861 0.764 1.174 0.774 0.696** 1.139 0.695* 0.587*** 0.946 (0.151) (0.133) (0.199) (0.140) (0.122) (0.224) (0.151) (0.112) (0.205) household controls yes yes yes yes yes yes school controls & FE yes yes yes Observations 1,978 1,978 1,978 1,978 1,978 1,978 1,978 1,978 1,978 Note: 1 = not in school, doing unpaid work only 2= not in school and doing some paid work 3 = in school, odds ratios are presented relative to base category of neither in school nor working. Sample is restricted to those who are enrolled in school in period 1. Robust standard errors in parenthesis (standard errors are transofrmed using the delta method for relative risk ratios) *** p<0.01, ** p<0.05, * p<0.1 10 10/30/2016 Table 4A. Associating test scores in period 1 and mental health in period 2 (1) (2) (3) Math and Verbal Test Score (pctile) 0.092*** 0.086*** 0.075*** (0.021) (0.021) (0.022) received scholarship 0.590 1.657 1.508 (1.080) (1.308) (1.396) age -1.224*** -1.279*** -1.419*** (0.393) (0.384) (0.505) male 4.230*** 5.607*** 5.336*** (1.197) (1.466) (1.639) household controls YES YES school controls & FE YES Observations 2,788 2,788 2,788 R-squared 0.017 0.025 0.050 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 11 10/30/2016 Table 4B. Associating test scores in period 1 and mental health in period 2 (1) (2) (3) Math and Verbal Test Score (pctile) 0.055** 0.052** 0.042* (0.021) (0.021) (0.021) Period 1 mental health (pctile) 0.213*** 0.211*** 0.212*** (0.017) (0.017) (0.016) received scholarship -0.227 0.636 0.578 (1.076) (1.300) (1.365) age -0.902** -0.968** -1.139** (0.383) (0.378) (0.486) male 3.875*** 5.070*** 4.806*** (1.168) (1.484) (1.645) household controls YES YES school controls & FE YES Observations 2,788 2,788 2,788 R-squared 0.061 0.068 0.092 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 12 10/30/2016 Table 5A. Test score in period 1 and subjective social status relative to community in period 2- current perceived position versus future expectation Social ladder relative to community (scale from 1-11) (1) (2) (3) Current Future Current Future Current Future Math and Verbal Test Score (pctile) 0.001 0.007*** 0.001 0.007*** 0.001 0.007*** (0.001) (0.001) (0.001) (0.001) (0.001) (0.002) received scholarship -0.171*** 0.077 0.037 0.175* 0.067 0.205* (0.060) (0.071) (0.067) (0.092) (0.069) (0.105) age 0.026 -0.022 0.030* -0.018 0.023 -0.041 (0.018) (0.025) (0.017) (0.024) (0.022) (0.025) male -0.137* 0.201* 0.034 0.287** 0.028 0.306** (0.071) (0.101) (0.074) (0.121) (0.083) (0.128) household controls YES YES YES YES school controls & FE YES YES Observations 2,788 2,777 2,788 2,777 2,788 2,777 R-squared 0.008 0.016 0.049 0.025 0.071 0.057 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 13 10/30/2016 Table 5B. Test score in period 1 and subjective social status relative to community in period 2 - current perceived position versus future expectation (controlling for period 1 mental health) Social ladder relative to community (scale from 1-11) (1) (2) (3) Current Future Current Future Current Future Math and Verbal Test Score (pctile) 0.000 0.006*** 0.000 0.006*** -0.000 0.006*** (0.001) (0.001) (0.001) (0.001) (0.001) (0.002) Period 1 mental health (pctile) 0.004*** 0.006*** 0.004*** 0.006*** 0.004*** 0.006*** (0.001) (0.001) (0.001) (0.001) (0.001) (0.001) received scholarship -0.187*** 0.056 0.017 0.148 0.049 0.179* (0.060) (0.070) (0.067) (0.092) (0.070) (0.106) age 0.032* -0.013 0.036** -0.010 0.029 -0.033 (0.018) (0.025) (0.017) (0.024) (0.022) (0.025) male -0.144** 0.192* 0.024 0.273** 0.018 0.291** (0.072) (0.101) (0.074) (0.120) (0.084) (0.128) household controls YES YES YES YES school controls & FE YES YES Observations 2,788 2,777 2,788 2,777 2,788 2,777 R-squared 0.014 0.024 0.054 0.032 0.078 0.065 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 14 10/30/2016 Table 6A. Test score in period 1 and subjective social status relative to Cambodia in period 2- current perceived position versus future expectation Social ladder relative to Cambodia (scale from 1-11) (1) (2) (3) Current Future Current Future Current Future Math and Verbal Test Score (pctile) -0.007*** -0.004*** -0.007*** -0.004*** -0.008*** -0.004*** (0.001) (0.001) (0.001) (0.001) (0.001) (0.001) received scholarship -0.071 0.001 -0.009 0.020 -0.000 0.028 (0.047) (0.069) (0.058) (0.089) (0.064) (0.104) age 0.050*** 0.014 0.054*** 0.015 0.066*** 0.007 (0.017) (0.023) (0.017) (0.023) (0.020) (0.026) male -0.106 0.046 -0.058 0.064 -0.046 0.096 (0.065) (0.079) (0.073) (0.104) (0.078) (0.109) Household and Family Controls YES YES YES YES School Fixed Effects YES YES Observations 2,786 2,775 2,786 2,775 2,786 2,775 R-squared 0.025 0.004 0.031 0.007 0.058 0.038 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 15 10/30/2016 Table 6B. Test score in period 1 and subjective social status relative to Cambodia in period 2 - current perceived position versus future expectation (controlling for period 1 mental health) Social ladder relative to Cambodia (scale from 1-11) (1) (2) (3) Current Future Current Future Current Future Math and Verbal Test Score (pctile) -0.008*** -0.005*** -0.008*** -0.005*** -0.008*** -0.005*** (0.001) (0.001) (0.001) (0.001) (0.001) (0.002) 0.003*** 0.004*** 0.003*** 0.004*** 0.003*** 0.004*** Period 1 mental health (pctile) (0.001) (0.001) (0.001) (0.001) (0.001) (0.001) -0.083* -0.013 -0.024 0.002 -0.014 0.011 received scholarship (0.047) (0.070) (0.057) (0.090) (0.064) (0.105) 0.054*** 0.019 0.058*** 0.020 0.070*** 0.013 age (0.017) (0.022) (0.017) (0.022) (0.020) (0.025) -0.111* 0.040 -0.065 0.055 -0.053 0.086 male (0.065) (0.078) (0.072) (0.103) (0.077) (0.109) Household and Family Controls YES YES YES YES School Fixed Effects YES YES Observations 2,786 2,775 2,786 2,775 2,786 2,775 R-squared 0.029 0.008 0.034 0.011 0.062 0.042 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 16