What is a holistic score?
A holistic score is an overall score (usually on a 4- or 6-point scale) that is given to an essay. The Criterion holistic scoring compares a student’s writing to thousands of essays written and evaluated by writing instructors.
The essays used to build the scoring models have been scored by trained readers and were written by students under timed testing conditions. The writers had no opportunity to revise, use a spell checker or reflect on what they had written. So when students write on the Criterion topics in a regular class, working under more relaxed conditions, instructors and students should recognize that students’ scores may not precisely compare to those of the samples.
What are trait level indicators?
The Criterion service includes the option of reporting trait level indicators in addition to the holistic score and diagnostic feedback currently reported in the application. Individual trait level indicators can be enabled or disabled by the instructor on the Create Assignment screen.
The three traits for which information is provided are:
- Grammar, Usage and Mechanics (grouped together for effective trait level feedback)
- Style
- Organization & Development
If a trait receives more errors/comments than expected for a holistic score, a message will display to the student, indicating that the trait category needs attention.
These messages serve as a clear signal when the proficiency level of one or more traits is not consistent with the holistic score received for that essay. The trait identified as needing attention is specific to that essay submission and is not necessarily indicative of the student’s writing in general.
For example, in an essay that receives a holistic score of 5 on a 6-point scale, the expectation for proficiency is that there will be relatively few errors in grammar, usage and mechanics, and few suggestions for improvements in the areas of style, and organization and development. We know this because the e-rater® scoring engine analyzes these components, among several others, as part of the computation of the holistic score. If, however, the essay includes more errors in grammar, usage and mechanics than is typical for essays that usually receive this score, then a trait feedback message will display. This information focuses attention on specific aspects of the essay where improvements are likely to have the most impact on the overall holistic score.
The diagnostic feedback also provided in the Criterion service can provide additional detail to help find and address weak areas in the essay.
How are holistic scores and trait level indicators related?
Any trait level indicator displayed is relative to the holistic score. It indicates that the proficiency level of the trait is not consistent with the holistic score received for that essay and is lower than expected.
How does the Criterion service come up with its scores?
The Criterion service is based on a technology called e-rater that was developed at Educational Testing Service. The e-rater scoring engine compares the new essay to samples of essays previously scored by faculty readers, looking for similarities in sentence structure, organization and vocabulary. Essays earning high scores are those with characteristics most similar to the high-scoring essays in the sample group; essays earning low scores share characteristics with low-scoring essays in the sample group. Naturally, the sample essays must be scored very carefully and the collection must include sufficient essays at each score point.
What is the technology used in e-rater scoring?
E-rater scoring is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using NLP for the past 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information.
E-rater scoring uses NLP to identify the features of the faculty-scored essays in its sample collection and store them — with their associated weights — in a database. When e-rater evaluates a new essay, it compares its features to those in the database in order to assign a score.
Because the e-rater scoring engine is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which the e-rater database is created.
Can students trick the Criterion service?
Yes. Since the e-rater engine cannot really understand English, it can be fooled by an illogical, but well-written, argument. Educators can stop students from deliberately trying to fool the Criterion service by announcing that a random sample of essays will be read by independent readers. The Criterion service will also display an “Advisory” along with the e-rater score when an essay displays certain characteristics that warrant attention compared to other essays scored against the same topic.
Must students be connected to the Internet to use the Criterion service?
Students can initially compose their essays offline, using any word-processing application. However, they will ultimately need an Internet connection to be able to cut and paste their essays into the Criterion essay submission box so their work can be scored and analyzed. For assignments that are timed, essays should be composed online only to ensure accountability by all students and to accurately reflect their writing skills in this environment.
Can I import student identifiers from my data management system?
Yes, the Criterion service has import capabilities at several levels.
A Criterion Administrator will use the <Advanced Import> feature to create college, course and class levels by importing the required data into the Criterion service from a comma-delimited (.csv) format.
Faculty will also be able to import student information into the Criterion service using the <Import Student Information> function on the Classes Report screen.
Details are provided in both the HELP text and the Criterion® User Manual and Administrator Supplement.
Can I save my data?
Yes, the Criterion service has both an <Export Report Data> and an <Archive Portfolios> feature that can be used to create export files in a comma-delimited format (.csv) that can be opened by most text editors and spreadsheet programs. Detailed instructions for both features are provided in the Criterion® User Manual and Administrator Supplement.
Back to Criterion FAQ Main