SGP Package Preparation

Prior to final SGP analysis of PARCC data, SGPstateData meta-data associated with PARCC must be finalized. The following list includes steps required prior to running SGP analysis.

• Note: Because of Fall testing and the potential to need to run Fall-to-Spring analyses, YEAR will be designated with YEAR.1 (Fall) and YEAR.2 (Spring). This potentially impacts the creation of Cutscores and Knots and Boundaries. In particular, the theta scale changes between Fall 2015 and Spring 2016 so theta based Cutscore and Knots and Boundaries are likely to differ when the scale changed.

• Final calculation of knots/boundaries for PARCC Scale Score and Theta values based following receipt of 2016 data. Note that the Fall 2014, Spring 2014, and Fall 2015 are on the same scale whereas the scale is change for Spring 2016. Knots and boundaries for thetas should be examined by YEAR to see whether they need to be distinguished. Values will be saved as PARCC_Knots_Boundaries.Rdata and placed in the Knots_Boundaries folder of SGPstateData for inclusion in SGPstateData

• Final calculation of achievement level cutscores (in the THETA metric) for 2016 and inclusion of those cutscores in SGPstateData. Note that Fall 2014, Spring 2014, and Fall 2015 are on the same scale whereas the scale is changed for Spring 2016. SGPstateData currently contains some theta cuts for old data (2014/2015).

• Final validation of configurations for SGP analysis will require establishing configurations to be use be used for SGP analyses. In particular, data must be examined to see whether any 2015_2016.1 to 2015_2016.2 analyses need to be conducted. Pearson will ideally provide 2015_2016.2 data (without scaled scores) in mid-June allowing for the investigation of configurations to be used.

PARCC Assessment Metadata stored in SGPstateData.

The use of higher-level functions included in the SGP package (e.g. analyzeSGP) requires the availability of state specific assessment information. This meta-data is compiled in a R object named SGPstateData that is housed in the SGP package. The required meta-data included a) knots and boundaries, proficiency level cutscores for each grade and content area combination, b) analysis configuration information and c) the norm group preferences object to guide instances in which students have multiple SGPs from different course progressions. The entry for PARCC as it appeared in 2015 can be seen here on Github.

Calculation and addition of knots and boundaries

Cubic B-spline basis functions are used in the calculation of SGPs to more adequately model the heteroscedasticity and non-linearity found in assessment data. These functions require the selection of boundary and interior knots. Boundary knots are end points outside of the scale score distribution that anchor the B-spline basis. These are generally selected by extending the range of scale scores by 10%. That is, they are defined as lying 10% below the lowest obtainable scale score (LOSS) and 10% above the highest obtainable scale score (HOSS). The interior knots are the internal breakpoints that define the spline. The default choice in the SGP package is to select the 20^th, 40^th, 60^th and 80^th quantiles of the observed scale score distribution.

In general the knots and boundaries are computed from a distribution comprised of several years of test data (i.e. multiple cohorts combined) so that any irregularities in a single year are smoothed out. Subsequent annual analyses use these same knots and boundaries as well. All defaults were used to compile the knots and boundaries for the PARCC assessments, and seperate values were calculated for the original scale used from Fall 2014 to Fall 2015 and the updated scale first used in Spring 2016. State specific knots and boundaries were also calculated for those analyses.

Proficiency level cutscores

Cutscores, which are set externally through standard-setting processes, are mainly required for student growth projections. Projections will be used for individual student reporting, but not for accountability purposes.

The performance (achievement) level metadata, such as labels and descriptions, were also updated to reflect the new PARCC standards.

Conditional standard errors of measurement (CSEMs)

The calculation of SIMEX adjusted SGPs and SGP standard errors require the availability of each assessments' standard errors of measurement. Because these values are not scale score specific (i.e. there are some identical test scores with different CSEM values) they are included in the student level data. If these values were scale score specific they would be added to the SGPstateData object. The use of student level values requires the identification of the name of the student level CSEM variable, "SCALE_SCORE_CSEM", in the metadata here.

Set the sgp.cohort.size SGP Configuration

Given the decreasing quality of model fit and the increasing difficulty in interpretability of what SGP values from small norm groups, the decision was made by the PARCC Technical Advisory Committee and consortium member states to institute a minimum N size of 1,000 fstudents or all analyses. This could be done either manually by removing the configuration scripts, or by adding a SGP_Configuration entry for it in the SGPstateData. The latter option was adopted, and therefore any future attempts to examine course progressions with fewer than 1,000 students will require a manual override of this (i.e. set to NULL).

SGPstateData[["PARCC"]][["SGP_Configuration"]][["sgp.cohort.size"]] <- 1000 # NULL to remove restriction

Norm group preferences

The process through which some EOCT analyses are specified can produce multiple SGPs for some students. In order to identify which quantity will be used as the students' "official" SGP and subsequently merged into the master longitudinal data set, a system of norm group preferencing is established and is encoded into a lookup table and included in the SGPstateData. In general, the preference is given to:

• Progressions with the largest cohort size.
• Progressions with the most recent prior score (i.e. progressions from Fall 2015 to Spring 2016 preferenced over progressions from Spring 2015 to Spring 2016).

For more information of the creation of the analysis configuration scripts used to create the norm group prefereces object, see the section on course progression identification.