Samples behind every panel

 

Sampling and Data Collection of the National Panel

National Panel consist of about 5,000 Russian schoolchildren are studied since 2011, when they were in the 8th grade. The pupils took part in tests and a survey, their parents and teachers were also interviewed. Data is collected every 1-2 years.

Phases of data collection within the national panel:

• TIMSS 2011 (8th grade students)
• First Wave - 2012 (9th grade students)
• PISA 2012 (9th grade students)
• Second wave - November-December 2013 (11th grade students or students of institutions of vocational education)
• Third Wave - April-June 2014 (11th grade students or students of institutions of vocational education)
• Follow-up CATI wave – August 2014

      
A unique feature of the national panel is that the same respondents participated in TIMSS 2011 and the International Program for Student Assessment (PISA 2012). In Russia TIMSS 2011 and PISA 2012 studies were carried out by the Center for Educational Quality Assessment, that is a part of the Institute of content and teaching methods of the Russian Academy of Education (ISMO RAO) with the participation of the Ministry of Education and Science of the Russian Federation, the Federal Service for Supervision in Education and Science, as well as regional education authorities.
The first, second and third waves of TrEC were conducted by a research company with the support of the Department of Education.

National Longitudinal panel of schoolchildren is based on a sample of students made within the international comparative study of mathematics and science education TIMSS (Trends in Mathematics and Science Study) in 2011 for 8th grade students in the schools of the Russian Federation. The sample was prepared by the Center of Educational Quality Assessment ISMO RAO and approved by the International Coordination Centre for TIMSS Research (ISC - International Study Center, Boston College, USA).

 

а)  Sample description of TIMSS 2011 wave[1]

In TIMSS 2011 study students of 4th and 8th grades were the object of study and their mastering of curriculum was estimated by the end of 4th and 8th grades in particular. Next we will describe the sample of schoolchildren of 8th grade, as it became the basis of TrEC study sample, i.e. in the subsequent waves the same study participants were interviewed as that taking part in TIMSS 2011.

Within TIMSS 2011 study the knowledge of 8th grade schoolchildren in math and science was estimated, in addition, the surveys of students, their teachers (mathematics and natural science subjects) and the school administration were held.

The sample is representative of 8th grade in 2010-2011 in the Russian Federation. When selecting at the school level very small schools and schools for children with special needs were excluded. In schools children with special needs and children who do not speak Russian were excluded from the sample.

In Russia the formation of the sample was conducted in two stages: selection of regions and selection of schools. In constructing the sample of regions the number of students in the region was taken into account (based on the methodology of probability sampling proportional to the volume (PPS, Probability Proportionate to Size Sampling)). The largest regions were included in the sample with probability = 1.

In all the regions in the sample a list of schools in the region was created. The selection of schools was carried out taking into account the size of the population aggregate. In each selected region 4 to 12 educational institutions were selected.

The sample included only schools with Russian as the language of instruction. Each school selected one or two classes, all the students in them participated in the testing.

The overall number of participants of the testing is 4893 students attended 8th grade in 210 schools in 42 regions of the Russian Federation. The study sample by region is presented in Table 1.

Data collection for TIMSS 2011 study was carried out via regional and school coordinators under the leadership of the Center for Educational Quality Assessment ISMO RAO. Data collection method consists of filling in test materials and questionnaires by students in the classrooms.

 

b)  Data collection and sampling of the first wave of the national panel

As part of the first wave of the national panel for the 2011-2012 academic year, the surveys were conducted among TIMSS 2011 participants and their parents (mothers). The survey collected data on educational and labor trajectories of the study participants, who at that time were students of 9th grade.

Field work was carried out by "NAFI company" with the involvement of regional coordinators who participated in the TIMSS 2011 study. Survey process was organized as follows: the regional coordinators contacted the school with a request to participate in the study and to assign a responsible employee. Students participating in TIMSS 2011 got an  envelope with the questionnaire for students and one more for parents plus a paper consent to participate in the study ( for under aged schoolchildren). Students passed completed questionnaires in the envelopes to responsible employees in schools and he / she gave them to the interviewer.

For sampling data of TIMSS 2011 in the form of a list of regions, communities and schools was used. During the first wave of the national panel 3827 students in 9th grade in 42 regions of the Russian Federation were interviewed. Number of respondents of the first wave of the TrEC by region is presented in Table 1.

Table1. Number of respondents of TIMSS 2011 study and of the first longitudinal wave

№	Region	Number of schools in TIMSS 2011 sample	Number of schoolchildren interviewed		Number of first wave participants that were in fact interviewed in TIMSS 2011	Response rate
			TIMSS 2011	First wave  2012
1.	Altai Krai	4	80	77	67	84
2	Amur Region	4	83	66	64	77
3.	Astrakhan Region	4	111	98	70	63
4.	Bryansk region	4	85	76	70	82
5.	Volgograd Region	4	113	102	99	88
6.	Vologda Region	4	87	58	55	63
7.	Voronezh Region	4	108	79	77	71
8.	Moscow	12	284	191	176	62
9.	St. Petersburg	6	151	39	38	25
10.	Trans-Baikal Territory	4	105	69	64	61
11.	Irkutsk Region	4	83	30	30	36
12.	Kaliningrad Region	4	118	81	72	61
13.	Kaluga Region	4	78	82	48	62
14.	Kemerovo  Region	4	130	102	98	75
15.	Kostroma Region	4	100	79	74	74
16.	Krasnodar  Krai	8	170	166	143	84
17.	Krasnoyarsk Krai	6	130	114	67	52
18.	Kursk Region	4	128	74	73	57
19.	Moscow Region	10	222	135	126	57
20.	Nizhny Novgorod Region	6	135	137	112	83
21.	Novrogod Region	4	83	89	66	80
22.	Novosibirsk Region	4	113	89	66	58
23.	Orenburg Region	4	75	55	45	60
24.	Perm Krai	4	140	110	102	73
25.	Republic of Bashkortostan	8	167	158	150	90
26.	Republic of Dagestan	8	164	130	121	74
27.	Republic of Sakha (Yakutia)	4	92	47	41	45
28.	Republic of North Ossetia (Alania)	4	103	98	90	87
29.	Republic of Tatarstan	6	130	124	123	95
30.	Rostov Region	6	126	62	50	40
31.	Ryazan Region	4	77	77	72	94
32.	Samara Region	6	132	111	53	40
33.	Saratov Region	4	94	89	87	93
34.	Sverdlovsk Region	6	141	120	118	84
35.	Stavropol Krai	4	114	86	81	71
36.	Tomsk Region	4	86	62	61	71
37.	Tula Region	4	83	64	50	60
38.	Ulyanovsk Region	4	78	77	75	96
39.	Khanty-Mansi Autonomous Area	4	85	59	55	65
40.	Chelyabinsk Region	6	130	110	101	78
41.	Chuvash Republic	4	69	66	64	93
42.	Yamal-Nenets Autonomous Area	4	110	89	83	75
TOTAL		210	4893	3827	3377	69

Reasons for non-participation in the 1^st longitudinal wave:

1. Refusal to participate at the level of school authorities. in Moscow Region (2 schools), St. Petersburg (3 schools), Rostov region (1 school).

2. Refusal to participate from some students or parents (negative attitude of parents to the survey).

3. Inaccessibility for participation because of the geographical remoteness of some schools , the weather conditions at the moment of data collection (the regions of Western and Eastern Siberia).

4 Inaccessibility because of the mobility of the respondents (changing schools or changing of the place of residence).

 

It should be noted that during data collection the list of names of participants TIMSS 2011 was not available and in addition, some classes have been reorganized, thus in some schools students that didn't participated in TIMSS 2011 were interviewed as well. To evaluate the accessibility of TIMSS 2011 participants a comparison of TIMSS 2011 sample and that of the first longitudinal wave was undertaken. The comparison was done based on the names of the participants and in cases where this information was not available - based on sex, date of birth and other characteristics. 3435 respondents out of the total of 3827 participated in the first wave of the longitudinal panel were identified as respondents of TIMSS 2011. Thus, the response rate for the survey can be estimated as 69%. Characteristics of the sample by region (number of participants in TIMSS 2011, the first wave of the longitudinal panel, the response rate) are shown in Table 1. As can be seen from the table, the percentage of response rate varies from 25% (St. Petersburg) to 96% (Ulyanovsk Region). This range can be explained by non-response reasons listed above as well as by the quality of the regional coordinators work.

 

c) Data collection in the second and third longitudinal waves

After the first longitudinal wave, some changes in organization of fieldwork have been implemented in order to increase the response rate. The second and the third wave took part in the autumn of 2013 and spring of 2014, when study participant were 11 grade students (last grade in the school) or vocational school students (as some part of students leave general school after the 9^th grade and go to vocational school).

First of all, additional methods to track students with the administrative data were implemented. Every school was sent a form with a list of the previous wave participants and asked to fill it with the information about their current place of studying. It was possible as schools normally collect information about new place of studying of a student who left it before 11^th grade. As a result, apart from questionnaire, for more than 90% of students information about their educational track after the 9^th grade was gathered.  

This was later used by fieldwork managers and interviewers in order to find as many respondents as possible. In addition, in every region a coordinator was appointed by region authorities who helped interviewers to communicate with schools and get an access to students.

Finally, additional methods of data collection were used. During the first stage of the fieldworks, all study participants filled the questionnaires in the classrooms. Those who were not present in school were interviewed by telephone.  At the second stage, CATI and CAWI were used for those who were not found during the first stage, especially for those who moved to a different town or region.

Response rates for the second and third waves are the following:

•         Second wave (N=4138)   85%

•         Third  wave (N=4236)     87%

 

d) Weights for longitudinal waves

TIMSS study contains post-stratification weights for observations in order to ensure greater representativeness of the sample in relation to the population of the relevant age cohort in Russia (variable "houwgt"). The first wave of the longitudinal panel is based on a sample of TIMSS study but differs from it by the number of participants of TIMSS study (31%) that did not take part in the longitudinal panel. Thus, our task is to create additional weights that will correct potentially non-random selection of respondents in the first wave of the panel sample in comparison with a sample of TIMSS (creation of weights is described in more detail in the documentation for the database).

The weighting coefficients are created only for those respondents who participated in TIMSS (3377 students). For respondents who took part in the first wave of the panel but not in TIMSS study (450 people) the weights are not available. Thus, researchers have a choice between using a shortened weighted sample and more complete unweighted one. In practice, results of analysis in both cases are similar with a slight difference of 1-2%.

The weighting coefficients are also created for waves 2 and 3 to correct non-response compared to TIMSS 2011 study.

 

 

---

[1]  The description is based on the analytical report "The main results of an international study Mathematics and Science Education TIMSS-2011",

webpage: http://www.centeroko.ru/timss11/timss11_pub.htm, also see TIMSS report “Characteristics of National Samples” - T07_ TR_AppendixB,  p.409).