Understanding Organization and Development Feedback
How does the automated system make decisions about text segments in a student essay and the corresponding organizational labels?
The algorithm developed to automatically identify essay-based organizational elements is based on samples of faculty-annotated essay data. Two faculty readers were trained to annotate essay data with appropriate organizational labels. Readers labeled approximately 1,400 essays on college-level topics.
What is the agreement rate between two faculty readers on the labeling task?
Two faculty members are in general agreement on all labeling tasks.
What is the agreement rate between the system and the faculty reader?
The faculty reader’s assessment is in general agreement with the system. In the vast majority of cases, ETS researchers generally found either exact or adjacent agreement (within one point) between the Criterion scores and those of a trained essay reader. Both used the same scoring guidelines and scoring system.
Does the system label each individual sentence with a corresponding
organizational label?
Yes. Sometimes multiple sentences are associated with a single organizational element, and the entire block of text is highlighted and appears to be assigned a single label. In fact, each sentence is labeled individually.
Does the system label according to sentence position only?
No. Many features, including word usage, rhetorical strategy information, possible sequence of organizational elements and syntactic information are used to determine the final organizational label.
Back to Criterion FAQ Main