Experiment Report¶
Overall descriptive feature statistics¶
These values are reported before transformations.
Prevalence of recoded cases¶
This sections shows the number and percentage of cases truncated to mean +/- 4 SD for each feature.
Feature value distribution¶
The following table shows additional statistics for the data. Quantiles are computed using type=3 method used in SAS. The mild outliers are defined as data points between [1.5, 3) * IQR away from the nearest quartile. Extreme outliers are the data points >= 3 * IQR away from the nearest quartile.
Feature distributions and inter-feature correlations¶
Training set distributions¶
The following plot shows the distributions of the feature values in
the training set, after transformation (if applicable), truncation
and standardization (if applicable). The line shows the kernel density estimate. The
human score (sc1) is also included.
Response length (length) is included if you specified length_column in the config file, unless
the column had missing values or a standard deviation <= 0.
Inter-feature correlations¶
The following table shows the Pearson correlations between all the training features
after transformation (if applicable), truncation and standardization (if applicable). The human score
(sc1) is also included.
Response length (length) is included if
you specified length_column in the config file, unless the column had missing
values or a standard deviation <= 0.
The following values are highlighted:
- inter-feature correlations above 0.7, and
sc1-feature correlations lower than 0.1 or higher than 0.7
Marginal and partial correlations¶
The plot below shows correlations between truncated and standardized (if applicable) values of each feature against human score. The first bar (Marginal) in each case shows Pearson's correlation. The second bar (Partial - all) shows partial correlations after controlling for all other variables. If you specified length_column in the config file, a third bar (Partial - length) will show partial correlations of each feature against the human score after controlling for length. The dotted lines correspond to r = 0.1 and r = 0.7.
Description of the data¶
Responses excluded due to non-numeric feature values or scores¶
The rest of this report is based only on the responses that were not excluded above.
Model¶
Standardized and relative regression coefficients (betas)¶
The relative coefficients are intended to show relative contribution of different feature and their primary purpose is to indentify whether one of the features has an unproportionate effect over the final score. They are computed as standardized/(sum of absolute values of standardized coefficients).
Negative standardized coefficients are highlighted.
Note: if the model contains negative coefficients, relative values will not sum up to one and their interpretation is generally questionable.
Here are the same values, shown graphically.
Principal component analysis¶
PCA using scaled data and singular value decomposition. This is computed using processed features after the truncation of outliers and other transformations specified in feature config file.
Evaluation results¶
Overall association statistics¶
Descriptive holistic score statistics¶
The table shows distributional properties of human and system scores. SMD values lower then -0.15 or higher than 0.15 are highlighted.
Please note that for raw scores, SMD values are likely to be affected by possible differences in scale.
Association statistics¶
Confusion matrix¶
Distribution of human and machine scores¶
Additional fairness analyses¶
This section presents additional fairness analyses described in detail in Loukina et al. (2019). These analyses consider separately different definitions of fairness and can assist in further trouble-shooting the observed subgroup differences. The evaluations focuses on three dimensions:
Outcome fairness measures:
Overall score accuracy: whether the automated scores are equally accurate for each group. The metric shows how much of the variance in squared error $(S-H)^2$ is explained by subgroup membership.
Overall score difference: whether the automated scores are consistently different from human scores for members of a certain group. The metric shows how much of the variance in actual error $S-H$ is explained by subgroup membership.
The differences in the outcome fairness measures might be due to different mean scores (different score distributions) across subgroups or due to differential treatment of different subgroups by the scoring engine or both.
Process fairness measures:
- Conditional score difference: whether the automated scoring model assigns different scores to members from different groups despite them having the same construct proficiency. The metric shows how much additional variance in actual error ($S-H$) is explained by subgroup membership after controlling for human score, which can be thought of as a reasonable proxy for proficiency.
The differences in process fairness measures indicate differential treatment of different subgroups by the scoring model.
Links to intermediate files¶
Click on the hyperlinks below to see the intermediate experiment files generated as part of this experiment.