ANALYSIS OF THE EMPIRICAL STUDIES INVESTIGATING THE ETCH
ETCH Author: Susan J. Amundson, Ph.D., OTR/L, FAOTA
October 2004

At the end of 1995, the Evaluation Tool of Children's Handwriting (ETCH) came off the press. Since that time, ETCH test kits have traveled from our office in Homer, Alaska to many school districts, pediatric occupational therapy agencies, and children's hospitals throughout the U.S. and Canada and to far off places such as Tasmania and Turkey. Pediatric occupational therapists and child specialists are now implementing the ETCH with children struggling to write in every day life.
When the ETCH was published, it was "under construction" and lacked important psychometric information. Fortunately, since that time to the present, test reliability, validity, and normative data of the ETCH have been investigated through various research studies, mainly those of doctoral and master-level students. Based on the efforts of numerous occupational therapy practitioners, students, and professors, the ETCH itself has evolved through scientific investigation. The work of these researchers continues to contribute to the pool of information about the ETCH and to our knowledge base about children with handwriting difficulties. I salute and thank these researchers.
A synopsis of the empirical studies focused on the ETCH and my comments about each study and its implication for pediatric occupational therapy practice follows. My comments are italicized.

RELIABILITY STUDIES WITH THE ETCH

Diekema, S.M., Deitz, J., & Amundson, S.J. (1998). Test-retest reliability of the Evaluation Tool of Children’s Handwriting – Manuscript. American Journal of Occupational Therapy, 52, 248-255.

The primary purpose of this study was to examine the test-retest reliability of the legibility portion of the Evaluation Tool of Children’s Handwriting – Manuscript (ETCH-M). The tool evaluates six areas of children’s handwriting including: (a) alphabet production in uppercase and lowercase from memory, (b) numeral writing (1 through 12) from memory, (c) near-point copying, (d) far-point copying, (e) dictation, and (f) sentence composition. Thirty-one grade 1 and grade 2 students with handwriting deficits, as identified by a teacher or therapist in the Seattle metropolitan area participated. The ETCH-M was administered twice, 1 week apart, to each participant by the primary investigator who also served as the sole rater. The primary investigator was blind to the student identities and whether the score sheet originated from a test or retest administration.

Moderate reliability coefficients (intraclass correlation) were 0.77 for total letter legibility, 0.71 for total word legibility, and 0.63 for total numeral legibility. Reliability coefficients for letter legibility of individual tasks were generally lower and ranged from 0.20 (near-point copying) to 0.76 (alphabet uppercase writing). ETCH-M test-retest reliability, especially with individual tasks, was lower than desirable but within the range of other assessments measuring children’s handwriting. Possible reasons for unstable test scores may be a ceiling effect reached, the inherent subjectivity of scoring handwriting even with specific legibility criteria, and the high variability of handwriting performance in children with less stable sensorimotor systems.

Findings indicate that total legibility scores on the ETCH-M are more stable than individual task scores for reporting children’s handwriting function. Caution should be used when reporting legibility percentages for individual tasks of the ETCH-M. The authors recommend that not only legibility scores be used during handwriting evaluation but other factors such as writing speed, the impact of legibility components, and classroom observations be considered.

As with many pediatric occupational therapy assessments, the individual task scores have lower test-retest reliability than the total test scores. The ETCH is no different. This study directs occupational therapy practitioners evaluating children’s handwriting to not solely focus on ETCH-M legibility scores. A comprehensive handwriting evaluation needs to include an assessment of contextual features, a global view of what the child’s handwriting looks like, classroom observations and norms, the child’s handwriting speed, and the child’s handwriting within the context of the language arts writing process.

Schuette, J. (2001). Test retest reliability of the Evaluation Tool of Children’s Handwriting in assessing typically developing six to eight year olds. Unpublished master’s thesis, New York University, New York.

This study examined the test-retest reliability of the ETCH-M with six to eight year old children who were typically developing. Nineteen children in grade 1 and grade 2 at a private elementary school in Brooklyn, NY participated. None were receiving special therapeutic services (e.g., OT, PT, SLP). The primary investigator administered the ETCH-M twice to each participant with a 7 to 8 day testing interval. Response booklets were scored by a sole rater using the ETCH-M legibility scoring guidelines.

High test-retest reliability coefficients for total legibility scores were found: 0.95 for total word, 0.88 for total letter, and 0.84 for total numeral. Stability was lower for letter legibility of individual tasks ranging from 0.11 (sentence composition) to 0.95 (uppercase letter writing). Individual task reliability for word legibility ranged from 0.64 (sentence composition) to 0.81 (far point copying). All coefficients were reported as interclass correlation coefficients. Analyses of the types of error in alphabet writing indicated that most of the errors noted were not due to legibility/letter formation but due to letter omission and letter case mistakes.

This study’s results compared to the above study by Diekema, Dietz, and Amundson (1998) indicate that the test retest reliability of the ETCH-M with children who are typically developing were more stable over time on total legibility scores. By including children who were typically developing, this study was able to assess the stability of the test itself as opposed to the stability of the children. A limitation of this study was its small sample (n = 19).

Test retest reliability coefficients for total legibility scores in the Schuette study were high, in contrast to the Diekema et al. study that found moderate test retest reliability coefficients for total legibility scores. Schuette examined children who were typically developing whereas Diekema et al. assessed only children identified with handwriting problems. By reviewing both of these studies, we are able to examine the stability of the ETCH-M itself (Schuette) and the stability of the test combined with the stability of children with handwriting problems (Diekema et al.). When only test stability is examined, higher reliability of the ETCH is found. In clinical practice, we can expect to find greater variability amongst the performances of children with disabilities because of the unstable nature of their sensorimotor systems.

Grace-Frederick, L. (1998). Printing legibility, pencil grasp, and the use of the ETCH-M. Unpublished master’s thesis, Boston University.

One research question of this study focused on replicating the interrater reliability study of the ETCH-M and comparing its results to the interrater reliability findings reported in the ETCH Examiner’s Manual. Participants included 133 grade 2 students from six regular education classrooms in the Boston metropolitan area. Ten of the children had been referred to occupational therapy services in the past 2 years. One practitioner tested all of the children. Recommended ETCH-M directions to assess each child individually were not used. Instead children were evaluated as a classroom group and administration procedures modified to do so. ETCH-M response booklets were scored by three experienced pediatric occupational therapists.

Interrater reliability intraclass correlation coefficients (ICCs) for total legibility were 0.74 for letters, 0.89 for numerals, and 0.50 for words. ICCs for the individual tasks ranged from 0.47 through 0.96 with six of ten tasks being higher than the original study reported in the ETCH Examiner’s Manual. Percentage-of-agreement based on 1-point ranged from 76.7% to 100.0% on individual tasks. Interrater reliability results of this study ranged from moderate to good.

Although reliability among raters was higher in this study than the original study reported in the ETCH Examiner’s Manual, the participants in the original study included only children who had been referred or were receiving occupational therapy services. Because the children in the Grace-Frederick study were all in regular education with the exception of only 10 children having been referred to occupational therapy, the amount of legibility errors was undoubtedly less than the children in the original study. More legibility errors would occur with children who had recognized handwriting problems and would mathematically increase the chance for error in rater scoring as in the original study.

VALIDITY STUDIES WITH THE ETCH

Grace-Frederick, L. (1998). Printing legibility, pencil grasp, and the use of the ETCH-M. Unpublished master’s thesis, Boston University.

Another question of the Grace-Frederick study (see above) centered on the criterion-related validity of the ETCH-M to determine if the tool could discriminate between above average and below average handwriting groups. Participants were grade 2 students from six regular education classrooms. Each classroom teacher was asked to rate the daily printing legibility of each student. A 5-point rating scale was employed with "1" being much less than average, "2" being less than average, "3" being average, "4" being a little better than average, and "5" being much better than average. All participants were assessed with the ETCH-M by the same examiner.

Of the 133 participants, 33% (n = 44) received below average handwriting ratings (scores of 1 and 2) whereas 29% (n = 38) received above average handwriting ratings (scores of 4 and 5) as perceived by their classroom teachers. Children in the average range receiving a "3" constituted 38% (n = 51) and were not included in the data analysis. Using between group t-tests, the ETCH-M was able to discriminate between the less than average and the better than average handwriting of grade 2 students on three total legibility scores (letters, words, numbers) and 8 of 11 individual task scores.

Overall, the ETCH was able to discriminate between groups of children in second grade with less than average handwriting and better than average handwriting as measured by their teachers’ judgment. The findings indicate that the ETCH has an acceptable level of criterion-related validity for discriminating between second graders’ good and poor handwriting. The study would have been stronger had a reliability coefficient of the teachers’ scale been reported and a scoring methodology described.

Sudsawad, P., Trombly, C.A., Henderson, A., & Tickle-Degnen, L. (2000). The relationship between the Evaluation Tool of Children’s Handwriting and teachers’ perceptions of handwriting legibility. American Journal of Occupational Therapy, 55, 518-523.

The purpose of this study was to examine the social validity of the ETCH- M with teachers’ perception of children’s handwriting legibility. Participants included 45 grade 1 children who were identified by their teachers as having below average handwriting performance in the Boston metropolitan area. Each child’s teacher was asked to rate the child’s handwriting legibility for overall performance and for 5 specific writing tasks (i.e., near-point copying, far-point copying, dictation, free writing, and numeral writing during math) on a 7-point rating scale. Each child was tested with the ETCH-M individually. A sole scorer who was blind to the children’s identities scored the children’s response booklets.

No significant relationships between the ETCH-M and the teachers’ perception of the children’s level of handwriting legibility were found. Possible reasons cited for a lack of a significant relationship may be a discrepancy between what the ETCH-M measures and handwriting skills as judged by teachers, extraneous factors about children (e.g., lack of attention) entering into the teachers’ ratings, and the ETCH-M representing a one-time observation instead of day-to-day observation. The authors recommend replication of the study to verify their results.

The social validity of the ETCH-M is questioned in this study, however, the study itself lacks a stringent methodology to measure validity. No reliability is reported for the teachers’ rating scale for handwriting legibility. Furthermore, no procedure for the teachers’ rating methods was established. Did the teachers rely on their memories of each child’s handwriting legibility level in five different tasks? Did they observe each child performing various writing tasks and score the rating scale based on their observation? In addition, the teachers’ scale lacked face validity. With a 7-point scale, teachers were required to detect very discerning and nuanced differences of a student’s handwriting performance. For occupational therapists practicing in a school system, it would be surprising if a classroom teacher could discriminate between handwriting legibility levels or discern levels of a child’s poor handwriting from task to task (e.g., dictation vs. far-point copying) in the classroom. Although classroom teachers refer children to occupational therapy services for poor handwriting, few if any, distinguish poor handwriting from task to task.

Koziatek, S.M., & Powell, N.J. (2002). A validity study of the Evaluation Tool of Children’s Handwriting-Cursive. American Journal of Occupational Therapy, 56, 446-453.

This study investigated the concurrent validity of the ETCH-Cursive (ETCH-C) legibility scores with handwriting grades from teachers. Satisfactory and unsatisfactory handwriting performances were discriminated by legibility percentage scores of the ETCH-C. The participants were 101 grade 4 students in regular education with none of the participants receiving occupational therapy services. The primary investigator assessed each student individually with the ETCH-C and scored the response booklets randomly. Students also completed the Cursive Practice and Review work sheets at their school desks. The worksheets included the tasks of near-point copying, manuscript-to-cursive transition, and sentence composing. A three-teacher panel rated the students’ handwriting worksheets assigning an A, B, C, or unsatisfactory grade using established grading guidelines. Interrater reliability among the teachers was 0.91 (intraclass correlations coefficient).

Results revealed that the mean ETCH-C legibility percentage scores improved as the handwriting grade improved. A significant moderate relationship and ranking existed between the total cursive legibility percentage scores and the teacher-assigned handwriting grade. A moderate relationship between the legibility percentage scores of the ETCH-C and the classroom worksheets provides support that students demonstrated similar cursive handwriting performance in both settings. Scores discriminating between satisfactory and unsatisfactory handwriting were established through a chi-square analysis: 75% for ETCH-C total legible words score, 75% for classroom worksheet score, and 82% for ETCH-C total legible letters score.

Findings indicate a 75% total legible words score on the ETCH-C discriminated between unsatisfactory and satisfactory cursive handwriting by classroom teachers. Because the legibility score level ranged from 73% to 82%, the authors suggest that occupational therapy practitioners may want to reconsider the legibility level they use to begin intervention rather than rely on the level suggested in the ETCH Examiner’s Manual.

Although the legibility level for unsatisfactory handwriting was purported to be at the 85% level in the ETCH Examiner’s Manual, Koziatek and Powell’s study provides empirical evidence to refute this. Their study found that on the ETCH-C, discriminating legibility levels were at 75% for ETCH-C total word scores and 82% for total letter scores. Based on their study, I suggest that school-based practitioners use a range of scores from 73% to 82% as a benchmark to discriminate between readable and unreadable handwriting. As with all standardized tests, test scores can be deceiving and must be combined with other data, as in this case, data related to the child’s writing in the classroom. Determining appropriate educational and therapeutic services for a child must be considered within the context of the child’s global educational needs.

NORMATIVE DATA (SPEED AND LEGIBILITY) STUDIES

Addis, B. (1999). Handwriting legibility and speed ranges among first graders. Unpublished master’s thesis, Temple University, Philadelphia.

In this study, preliminary normative data for handwriting legibility and speed of first graders were gathered. Thirty children who were typically developing were tested individually with the ETCH-Manuscript during the last 3 months of their school year. Only the scores of three ETCH-M tasks (near-point copying, far-point copying, sentence composition) were analyzed.

Results indicated that 83.4% of the children had total word legibility percentages at 80% and above; 90.0% of the children had total letter legibility percentages at 80% and above; and that 100.0% of the participants had total numeral legibility percentages at 88% and above. All of the children scored at or above 75% of the total letter legibility at the end of grade 1.

First grade participants’ handwriting speed measured in letters per minute had great variability. See table below for specific means and ranges.

Occupational therapy practitioners frequently request handwriting speed and legibility norms for children. Addis’ study provides preliminary normative data of children at the end of grade 1 with a small sample (n = 30). Great variability is noted for both first graders’ handwriting speed and legibility percentages. Mean scores of handwriting speed assist the practitioner with determining if the referred child is at least keeping up with his or her peers. Again, data gathered with the ETCH must always be combined with a child’s contextual factors (e.g., classroom performance, grade level writing standards).

Koziatek, S.M. (2000). A validity study of the Evaluation Tool of Children’s Handwriting – Cursive in fourth grade students. Unpublished master’s thesis, Wayne State University, Detroit, Michigan.

In Koziatek’s study, Grade 4 children with 1.25 years of cursive instruction were assessed with the ETCH-Cursive (ETCH-C). Of these 101 children, 94% were in regular education, 6% received special education services, and none received occupational therapy services. Four writing tasks from the ETCH-C (i.e., near-point copying, far-point copying, manuscript-to-cursive transition, sentence composition) were examined. The primary investigator individually evaluated each student and scored the ETCH-C response booklets being blind to the students’ identities.

Results indicated that girls write faster than boys in Grade 4. As in the Addis’ study with first grade students (see above), great variability of handwriting speed amongst fourth graders also existed. Handwriting speeds ranged from 10 to 78 letters per minute for grade 4 students. See table below for specific means and ranges.

Via a one-way ANOVA, no significant relationship was found between mean letters per minute writing speed for the ETCH-C between grades for satisfactory and unsatisfactory handwriting (see above Koziatek and Powell study). The author reports that these results are not surprising because only a few studies have found a significant weak or inverse correlation between readability and speed.

Koziatek’s study offers preliminary normative data of the ETCH-C with fourth grade students. Although the samples are small and not demographically representative of the U.S., the wide range of handwriting speeds of the grade 4 students must be seriously considered when evaluating a child’s handwriting. Similar to Addis’ study, handwriting speeds have great variability amongst a grade level. Using ETCH speeds alone to determine a child’s need for occupational therapy would be a disservice to the child. Occupational therapists must use a commonsense approach combining standardized assessment results with other contextual factors of the child (e.g., attention span, language, spelling, syntax) that might interfere with the writing process.