Evaluating Student Writing with Language Models

Language models, which are computer programs designed to understand and generate text, have shown they can evaluate texts produced by machines. This study looks into whether these models can also effectively assess writing done by real people, especially students in a school setting. The goal is to see if these models can give helpful feedback to students trying to improve their writing skills.

#Purpose of the Study

The idea behind using language models for evaluating human writing is that they could provide quick and direct feedback. Good feedback can help students refine their abilities. However, human writing is often different from machine-generated text. For example, students might use words in unexpected ways. This difference can make it tricky to apply the same evaluation methods typically used for machine text to human-created work.

#Research Methodology

In this study, a total of 100 pieces of writing were collected from 32 Korean students. These writings included different types of compositions such as essays, reports, and scripts. The students were between 11 and 19 years old. The team used a specific language model, GPT-4-Turbo, to evaluate these texts based on five criteria: Grammaticality, Fluency, Coherence, Consistency, and Relevance.

The researchers provided feedback based on these evaluations and then asked students how they felt about the judgments. Were they reasonable, too harsh, or too lenient? This process helped determine how well the model could assess various writing styles.

#Results of the Evaluations

The evaluations showed that the language model was quite effective at judging grammaticality and fluency. In fact, the students agreed that the feedback on grammar was reasonable about 87% of the time and about 93% for fluency. However, the results were not as strong for the other three criteria. Students felt that evaluations of coherence, consistency, and relevance were sometimes off the mark, especially for more personal types of writing like diaries and self-introductions.

#Insights from the Findings

Though the results were not meant to be fully controlled or exhaustive, they offered some interesting insights. For instance, the language model tended to give higher scores for consistency and relevance, but lower scores for fluency. This suggested that the model might be a useful tool in helping students write more fluently.

Moreover, the evaluations for descriptive essays and book reports were generally favorable, indicating that the model could help students boost their writing scores. There was also a notable difference in average scores between younger and older students. Older students typically received higher scores, which hints that the model can differentiate between varying levels of writing skill based on age. This could be useful for helping younger students improve their writing.

#Related Research

Previous studies have focused on using different evaluation standards, like word matching or how similar a piece of writing is to a reference. However, using language models directly as evaluators has proven to be more effective in matching human grading, especially for machine-generated texts. Some studies have shown that using specific evaluation criteria tends to lead to more accurate and clear judgments.

This research builds on those ideas by applying them to human-written texts across multiple writing categories. By focusing on the strengths and weaknesses in students' writing, the goal is to improve their skills in a practical way.

#Evaluation Process

Gathering the writings for the study involved asking students to create their pieces without using any help from language models. Each submission came with specific writing instructions. The different types of writing included a wide range from reports to essays, ensuring a good mix of styles.

Once gathered, the texts were evaluated using the language model. The evaluation included giving scores from 1 to 5 based on how well each writing met the five criteria identified earlier. Each score came with feedback designed to highlight strengths and areas needing improvement.

#Checking the Feedback Validity

To see if the evaluations made sense, the researchers asked students to review the feedback and scores they received. They wanted to know if the students found the evaluations fair or if they felt they were viewed too harshly or kindly. Each student received payment for their participation, and while this part of the study had budget limitations, it was still a valuable way to gather perspectives on the feedback process.

#Overall Findings

The evaluations showed promising results. The language model provided reasonable assessments on 77% to 93% of the writing samples. This supports the idea that language models can be useful tools for identifying strengths and weaknesses in student writing.

There was a clear pattern in how the model performed. It scored higher on more objective types of writing like process essays and scientific reports. Meanwhile, for subjective types such as self-introductions and diaries, the feedback was considered less accurate. This suggests that while language models can help with many kinds of writing, they may not always be the best fit for evaluations that require a more nuanced understanding of personal expression.

#Age Differences in Writing

Another interesting point from the research was how the model ranked the writing of younger versus older students. In most cases, older students scored higher across the evaluation criteria. This suggests the language model can fairly judge the differences in writing skill that often come with age. The findings indicate that younger students might benefit particularly from using these evaluations to elevate their writing to match the standards of their older peers.

#Conclusion

This study expanded on the use of language models to assess human writing. By evaluating 100 different pieces of writing from a diverse group of students, it was shown that language models can accurately assess more objective writing aspects like grammar and fluency.

The research identified areas for improvement, particularly in subjective writing. Overall, the findings create a foundation for further exploration into how these tools can be effectively used in schools to help students become better writers. Future research could focus on refining evaluation methods and finding ways to help students directly revise their works based on the feedback they receive.

In the end, while language models show great promise as evaluators, the goal is to evolve these systems into effective, reliable aids for real-world writing improvement.