What’s Known

Assessment of handwriting using valid tools is crucial to provide objective measures and quantitative scores. Evaluation tools are developed to assess English, Chinese, and Hebrew writing, but these tools are not applicable to Iranian students.

What’s New

The evaluation of the construct validity by factor analysis led to two factors in the copying and dictation domain. Criterion validity revealed a low to moderate correlation. Acceptable internal consistency, excellent test-retest, excellent inter-rater between teachers, and good to excellent inter-rater reliability between teachers and the occupational therapist were reported. Therefore, the Persian Handwriting Assessment Tool was developed in Iran.

Introduction

Handwriting is a sophisticated neuromotor ability that involves complex cognitive and motor processes regulated by both central and peripheral processes’ hierarchical architecture. ^{1 , 2} According to the Occupational Therapy Practice Framework (OTPF), handwriting has an important role in learning and participation concepts; thus, the identification of handwriting deficits is of paramount importance. ^{2 , 3}

Handwriting problems, apart from orthographic problems are classified into perceptual, and motor types: The first is students’ inability to connect symbolic system and spelling, and the second is difficulty with movements necessary for writing in spite of speaking and reading abilities. These problems are seen in developmental coordination disorder children. ⁴ Primary school children spend up to 50% of school time performing writing tasks, some of which are performed under time constraints. ³ According to teacher estimates, a significant proportion of students struggle with handwriting difficulties. The prevalence of handwriting difficulties in school-aged children is 12%-13% and 23% in the UK and USA, respectively. Moreover, it was reported that 7.6% and 6.4% of Iranian boys and girls, respectively, had handwriting difficulty. ^{5 , 6} Difficulties in handwriting significantly affect students’ self-esteem, emotional well-being, academic achievement, and social functioning; therefore, early referral to occupational therapy is recommended to prevent these adverse effects. ^{2 , 5}

The assessment of handwriting using formal, standardized, and valid tools is crucial to provide objective measures and quantitative scores for clinical assessment and monitoring students’ handwriting performance progresses. ⁷ Students are expected to acquire handwriting proficiency during the first three years of school; thus, the best time to screen handwriting performance is grade two and the first half of grade three. ^{8 , 9} Evaluation tools in English, Chinese, and Hebrew language are not applicable to assess Iranian students’ handwriting performance. ^{9 - 12} Handwriting is a language- and culture-related skill; accordingly, any evaluation tool developed for Iranian students should suit their unique writing method of letters based on their position in a word and word formation. ^{13 , 14}

Havaei and colleagues developed Persian Handwriting Assessment Tool (PHAT) to assess handwriting legibility and speed in near-point copying and dictation domain in 8-10-year-old healthy students. Content validity ratio (CVR) was greater than 0.57. Exploratory factor analysis (EFA) revealed three components separately in copying and dictation domains. Confirmatory factor analysis (CFA) also confirmed the accuracy of the three-dimensional structure determined through EFA. Good to excellent internal consistency (0.84-0.99), intra-rater (0.87-1.00), and inter-rater reliability (0.7-1.00) were reported. ^{15 , 16}

The PHAT is a quickly-scored and the only Persian handwriting assessment tool available for Iranian students. Although the validity and reliability of the PHAT were investigated, comprehensive validation still needs to be established before it can be considered as a useful tool to be used by teachers, clinical application, and researchers for evaluating students’ performance in the future. Examining an instrument’s intrinsic factor structure will enhance measurement accuracy and lead to refinements within the instrument, thus improving its validity. ¹⁷

The current study was aimed to determine different types of validity and reliability of PHAT as to examine the following: construct and structural validity, criterion validity (concurrent validity), internal consistency, relative reliability (test-retest and inter-rater reliability), absolute reliability (standard error of measurement [SEM], and minimal detectable change [MDC)]).

Participants and Methods

The current study protocol was approved by the Ethical Committee of Iran University of Medical Sciences (ethical code: IR.IUMS.FMD.REC1396.9511355012). Permission to conduct the study in schools was obtained from the department of education in Tehran.

Participants

The participants were 452 healthy students aged 8-10 years (grade two, n=206; grade three, n=246). Students were selected based on the random cluster sampling method. The center of Tehran as a socioeconomically representative sample was selected. Two districts in the center of Tehran were randomly selected. Four governmental primary schools were randomly selected in each district. Students were randomly selected based on the inclusion criteria from grades two and three classes. Inclusion criteria were (a) students with no documented physical impairment that affects writing (orthopedic, neurologic, auditory, and visual); (b) students with no documented mental impairment based on the department of education test; (c) Persian was spoken as the primary language. Participants were excluded if they (a) had received special education; (b) were not willing to participate in the study. Informed consent was obtained from each participant’s parents. A code number was placed on students’ names to secure confidentiality. Sample size was calculated based on the rule of thumb. ¹⁸

Instrument

Havaei and colleagues developed PHAT to evaluate handwriting legibility and speed. PHAT requires the student to copy 12 words (خوشگل، نظافت، لطیفه، راضی، درس، ذهن، پنجره، صبح آغاز، مبعث، قوچ و کم) and dictate 12 words (آدم، اخلاق، بازی، عسل، صبحانه، قارچ، مریض، پژمرده، هفت، کاغذ، گنجشک، وطن) on a lined paper. PHAT evaluates five legibility components in copying and dictation domains: Space (space between letters and words), size (word size appropriateness), alignment (word angle on the line), slant (whole text angle on the line), and letter formation (correct ascending, descending, and rounding of letters). This tool is administered individually and requires 10 minutes to complete. ¹⁵

Assessment Protocol

PHAT was performed in a quiet and well-lit room. Each student was asked to sit behind a desk. Desk and chairs were appropriate based on the participant’s height to control the ergonomic factors. The required equipment was a pencil (HB lead), an eraser, a sharpener, a clipboard, a preprinted A4-size lined paper, and an anti-slip cover on the table to prevent extra movement of the clipboard. The clipboard was slanted for providing participants with a better pencil grasp and a parallel state of the forearm of the writing hand to the table. The students were asked to “read the words and then immediately write them using good handwriting”. The time each student took to complete the copy assignment was recorded. In the next step, the occupational therapist pronounced words loudly and expressively and gave the students enough time to write the words. ^{15 , 16}

Scoring Procedure

Time to perform the copying assignment (speed) was recorded and used to calculate the number of letters written per minute by the following formula: Number of Letters/Number of seconds=χ/60. Orthographic errors in dictation assignment were recorded. Speed and orthographic errors were scored within a wide range. Legibility components (word formation, spacing, alignment, text slant) were scored with a five-point Likert scale (ranging from very poor to very good), according to which five was considered as the best score. Size was scored differently (ranging from very small to very big), based on which three was considered as the best score. Finally, a mean score of twelve words for copy and dictation domains was considered as a given participant’s score.

Further Validity

Construct and structural validity were examined by conducting EFA and CFA, respectively. The minimum sample size to conduct factor analysis is 5-10 times more than the number of items of the instrument. Accordingly, separate samples of 200 and 252 students were used for EFA and CFA, respectively. ¹⁹ The factor structure of PHAT was examined by conducting an EFA using principal axis factoring method with Promax rotation. The Kaiser-Meyer-Olkin (KMO) test and Bartlett’s sphericity test were used to verify the suitability of both the research sample and the factor analysis model. The number of factors was calculated based on Eigenvalues, and objects with absolute load values of 0.3 or higher were deemed suitable. Several model fit indices and their criteria were used to investigate the goodness-of-fit of the model, including X² goodness-of-fit index per degree of freedom (2<X²/df<5; P>0.05), Root Mean Square Error of Approximation (RMSEA<0.8), Goodness-of-Fit Index (GFI>0.9), Adjusted Goodness-of-Fit Index (AGFI>0.9), Normed Fit Index (NFI), and Comparative Fit Index (CFI>0.9). ^{20 , 21} SPSS version 20 and Lisrel 8.8 (student version) were also used to analyze the data.

Criterion (concurrent) validity was examined by expert opinion as the gold standard. ²² Therefore, a teacher as an expert in this field was considered as the gold standard. A teacher with 28 years of teaching experience in grade two and three was asked to give a total score with a five-point Likert scale to 50 healthy students copy and dictation assignments with legibility components in mind. The correlation between the gold standard scores and the scores students obtained from PHAT scoring procedure was measured. Pearson correlation test was used. The strength of correlation was determined based on Munro’s guideline as 0.26-0.49 and 0.50-0.69 low and moderate correlation, respectively. ²³

Further Reliability

Internal consistency was examined using Cronbach’s alpha coefficient to test the internal consistency of the measurements of PHAT using a sample of 30 healthy students. Cronbach’s alpha is acceptable at a level of 0.70 or higher. ²⁴

Test-retest reliability was examined with a sample of 30 healthy students assessed twice with a seven-day interval based on similar articles. Diekema reported that short intervals between test and retest were appropriate for young children due to the developmental changes within a limited time frame. ²⁵

Inter-rater reliability (between teachers) was examined by involving 10 teachers with at least seven years of teaching experience in grade two and three and being aware of handwriting difficulties. Teachers with the mean of 20.6 years teaching experience scored scripts written by 50 healthy students in copying and dictation domain with the scoring procedure set in PHAT in the form of five groups (each group including two teachers). Each group scored scripts with the general agreement of two teachers. Also, inter-rater reliability was examined between each group of teachers and the occupational therapist.

Intra-class correlation (ICC) was used to calculate test-retest and inter-rater reliability. The ICC values above 0.75 and 0.6 to 0.74 indicate excellent and good reliability, respectively. ²⁶

The SEM is a reliability index that indicates the degree to which a score changes on repeated evaluations and is calculated as follows: SEM= SDpooled sqrt(1-ICC). As a rule of thumb, if SEM value is less than 10% of the maximum score of the scale, the level of absolute reliability is acceptable. ²⁶ The MDC, which is the minimal change that is beyond the measurement error in the score of an instrument, was calculated at 95% confidence interval using SEM values as follows: ²⁷ MDC 95=SEM*sqrt(2)*1.96.

Results

Descriptive Statistics

Out of 452 healthy students in grades two and three, 46.9% were boys. 45.6% of students were in grade two and 90.3% of students were right-handed. The mean age±SD was 9.04±0.54 years old. Table 1 presents the detailed demographic characteristics of the participants.

Further Validity

Construct and structural validity: The EFA was examined using principal axis factoring method with Promax rotation and a sample of 200 healthy students. The value of KMO and Bartlett’s test of sphericity confirmed sampling adequacy (copying: KMO=0.609, x²=249.868, df=6, P<0.001; dictation: KMO=0.595, x²=230.870, df=6, P<0.001). Two factors were identified separately for copying and dictation domains including global legibility (formation and space) and inclination (alignment and text slant) based on Eigenvalues greater than one (table 2). Speed (text speed per second and letter number per minute) in copying domain, orthographic error in the dictation domain, and size in both domains were considered as separate items due to different scoring procedures. These factors explained 64.40% for copying and 63.52% of the total variance.

The CFA was used to confirm the structural validity (factor structure) obtained from the EFA. The model fit indices for copy and dictation domains were: GFI=0.99, 0.99, AGFI=0.94, 0.96, NFI=0.99, 0.98, CFI=0.99, 0.99, and RMSEA=0.088, 0.064 in copy and dictation, respectively. Table 2 presents the goodness-of-fit indices and suggests that this model fits well. The path diagram for copying and dictation domains are provided in figures 1 and 2.

Figure 1. The measurement model indicates two dimensions in the copying domain of Persian Handwriting Assessment Tool.

Figure 2. The measurement model indicates two dimensions in the dictation domain of Persian Handwriting Assessment Tool.

Concurrent validity was examined using a sample of 50 healthy students (grade two, n=21; grade three, n=29). Low and moderate correlations in copying domain (formation: 0.548, P<0.001; spacing: 0.553, P<0.001; alignment: 0.442, P<0.001, size: -0.376, P=0.007; slant: 0.360, P=0.010) and dictation domain (formation: 0.503, P<0.001; spacing: 0.307, P=0.030; alignment: 0.358, P=0.011, size: -0.445, P<0.001; slant: 0.372, P=0.008) between the gold standard and scores obtained from PHAT scoring procedure were observed (table 3).

Further Reliability

Internal consistency was acceptable (Cronbach’s alpha: 0.72-0.99 and 0.74-0.99) for copy and dictation, respectively. Excellent test-retest (ICC: 0.76-0.91 and 0.76-0.99; SEM: 0.051-7.87 and 0.040-0.369; MDC: 0.142-21.75 and 0.110-1.019) and inter-rater reliability between teachers (ICC: 0.86-0.95 and 0.87-1.00; SEM: 0.064-0.148 and 0.062-0.130; MDC: 0.178-0.409 and 0.173-0.359) were reported for copy and dictation, respectively (table 4). Good to excellent inter-rater reliability between teachers and the occupational therapist (ICC: 0.63-0.91 and 0.6-0.95; SEM: 0.075-0.252 and 0.057-0.283; MDC: 0.208-0.696 and 0.158-0.783) was reported for copy and dictation, respectively (table 3).

Discussion

Assessment tools are needed to identify and support children with handwriting problems. The current study assessed the validity of PHAT in healthy 8-10-year-old native Iranian students. The results indicated that PHAT had two separate factors in copying and dictation domains including global legibility (formation and space) and inclination (alignment and text slant). Speed (text speed per second and letter number per minute) in copying domain, orthographic error in dictation domain, and size in both domains were considered as separate items due to different scoring procedures. Concurrent validity showed low and moderate correlations between the gold standard and scores obtained from PHAT scoring procedure. Acceptable internal consistency, excellent test-retest, and inter-rater reliability between teachers, good to excellent inter-rater reliability between teachers and the occupational therapist were reported.

Further Validity

The results of EFA in the current study indicated that PHAT was an instrument with two separate factors in copying and dictation domains (global legibility and inclination). The CFA model also confirmed the factor structure. Yong and Pearce stated that a factor with two variables happened only when the variables were highly correlated with each other. ²⁸ According to the literature, speed and legibility scores, as two important elements in handwriting performance, are reported separately. ²⁹ Speed was considered as a separate item. Li-Tsang and Havaei reported that speed was loaded in a different factor. ^{11 , 15} Word alignment and text slant in the two domains were loaded in a separate factor. Havaei reported that alignment and text slant in both domains loaded with other legibility components. ¹⁵ The reason for this separation can be explained through the development of handwriting, which begins with imitating vertical strokes and oblique crosses as writing readiness. A manuscript with poor alignment can still be legible while poor performance in other legibility components considerably reduces handwriting legibility. ²⁹ Word size in two realms and lexical error in the dictation domain were treated as separate items, and this was done in accordance with Havaei’s study, which stated that size in both domains and lexical error in the dictation domain were loaded into a separate component and clarified that this may have occurred as a result of the use of different scoring methods to measure these variables. ¹⁵

The assessment of the concurrent validity of PHAT showed low to moderate correlation between gold standard and scores obtained from PHAT scoring procedure. The obtained results were in accordance with those of Koziatek. ¹³ Experienced teachers were found as good judges of handwriting eligibility. ¹³ Koziatek and colleagues and Graham and colleagues reported that teachers placed a higher mark on appearance (word formation) when grading handwriting, which was in accordance with the results of the current study. ^{13 , 30} Moderate correlation between PHAT scoring procedure and the expert scores provided support for PHAT as a valid test to measure handwriting skills.

Further Reliability

The results of the present study on the reliability of PHAT revealed that it had acceptable internal consistency. This finding was in agreement with that of previous studies. ^{3 , 5 , 16} The point that during the process of the development of the tool, expert opinions were taken into consideration for word selection could be regarded as the reason behind the acceptable internal consistency. Li-Tsang reported moderate internal consistency for measurement items of Chinese handwriting analysis system and explained the result by the complex nature of handwriting activity. ¹¹

Items Measurement showed excellent test-retest reliability which was in line with the results of the previous studies. ^{3 , 11} This finding was significant when comparing the reliability of PHAT to that of other most full-length and in-depth handwriting assessments. ⁹

Legibility components showed excellent inter-rater reliability between teachers and good to excellent inter-rater reliability between teachers and occupational therapist. Hammerschmidt and Sudsawad concluded that teachers graded students’ handwriting based on their subjective judgment of handwriting quality (e.g., legibility, neatness, writing between the lines). ³¹ This finding was compatible with the findings of Rosenblum and colleagues and Ziviani and colleagues who reported that high agreement could be obtained between the ratings of experienced teachers for overall legibility. ^{9 , 32} Rosenblum and colleagues and Daniel and colleagues reported discrepancies between the inter-rater reliability of handwriting between teachers and occupational therapists and explained it through lenient ratings of teachers and their attention to the past performance of children when judging their performance. ^{9 , 33}

Values of MDC in test-retest, inter-rater between teachers, and inter-rater between teachers and occupational therapist help clinicians and researchers to decide on true performance changes over time.

Some of the strength of the study were a large sample size, governmental schools with a high number of classrooms, and average socioeconomic status for increased generalizability of the results. Test administration and scoring procedures were conducted by one person to maintain consistency.

Implications for Rehabilitation

The strength of validity and reliability coefficients of the current study helps teachers, clinicians, researchers, and school-based therapist to measure handwriting performance more confidently. Treatment decisions can be made based on the scores that students achieve. Researchers can perform outcome-based studies on handwriting remediation and treatment plans.

This tool helps us to cope with problems emerging when assessing handwriting, but it cannot be used as a diagnostic tool. In order to diagnose handwriting problems, we need a comprehensive assessment of educational and developmental history.

The current study had several limitations. Environmental noise and light in some schools, low co-operation of some school teachers and school officials, and the difficulty in determining the ergonomic and biomechanical features were the impediments the researchers encountered when conducting the study.

Conclusion

The results of the current study indicated that PHAT had acceptable validity and good to excellent reliability to assess legibility and speed. In order to identify at-risk or dysgraphic students, future studies on dysgraphic students are recommended to provide cut-off values, sensitivity, and specificity of the tool.

References

1. Wicki W, Hurschler Lichtsteiner S. Improvement of handwriting automaticity among children treated for graphomotor difficulties over a period of six months. J Occup Ther Sch Early Interv. 2018; 11:148-60. DOI
2. Falk TH, Tam C, Schellnus H, Chau T. On the development of a computer-based handwriting assessment tool to objectively quantify handwriting proficiency in children. Comput Methods Programs Biomed. 2011; 104:e102-11. DOI | PubMed
3. Rosenblum S. Development, reliability, and validity of the Handwriting Proficiency Screening Questionnaire (HPSQ). Am J Occup Ther. 2008; 62:298-307. DOI | PubMed
4. Martins MR, Bastos JA, Cecato AT, Araujo Mde L, Magro RR, Alaminos V. Screening for motor dysgraphia in public schools. J Pediatr (Rio J). 2013; 89:70-4. DOI | PubMed
5. Barnett AL, Prunty M, Rosenblum S. Development of the Handwriting Legibility Scale (HLS): A preliminary examination of Reliability and Validity. Res Dev Disabil. 2018; 72:240-7. DOI | PubMed
6. Shehni YM, Karami J, Shokrkon H, Mehrabizadeh Honarmand M. Prevalence of dysgraphia and the effects of multi-sense therapy on reduction of writing difficulties of primary school students in Ahvaz. Journal of Education and Psychology. 2004; 10:17-38.
7. Feder K, Majnemer A, Synnes A. Handwriting: current trends in occupational therapy practice. Can J Occup Ther. 2000; 67:197-204. DOI | PubMed
8. Overvelde A, Hulstijn W. Handwriting development in grade 2 and grade 3 primary school children with normal, at risk, or dysgraphic characteristics. Res Dev Disabil. 2011; 32:540-8. DOI | PubMed
9. Rosenblum S, Weiss PL, Parush S. Product and process evaluation of handwriting difficulties. Educ Psychol Rev. 2003; 15:41-81.
10. Feder KP, Majnemer A. Children’s handwriting evaluation tools and their psychometric properties. Phys Occup Ther Pediatr. 2003; 23:65-84. DOI | PubMed
11. Li-Tsang CW, Wong AS, Leung HW, Cheng JS, Chiu BH, Tse LF. Validation of the Chinese Handwriting Analysis System (CHAS) for primary school students in Hong Kong. Res Dev Disabil. 2013; 34:2872-83. DOI | PubMed
12. Roston KL, Hinojosa J, Kaplan H. Using the Minnesota Handwriting Assessment and Handwriting Checklist in screening first and second graders’ handwriting legibility. J Occup Ther Sch Early Interv. 2008; 1:100-15. DOI
13. Koziatek SM, Powell NJ. A validity study of the Evaluation Tool of Children’s Handwriting-Cursive. Am J Occup Ther. 2002; 56:446-53. DOI | PubMed
14. Standard databases for recognition of handwritten digits, numerical strings, legal amounts, letters and dates in Farsi languag. 23 October 2006. La Baule: Suvisoft. Tenth International Workshop on Frontiers in Handwriting Recognition.
15. Havaei N, Azad A, Zarei MA, Ebadi A. Development and Validity of the Persian Handwriting Assessment Tool for Primary School-Aged Children. Iran Red Crescent Med J. 2017; 19:e40508. DOI
16. Havaei N, Azad A, Alizadeh-Zarei M, Ebadi A. Reliability of Persian Handwriting Assessment Tool in Iranian primary school students. Iranian Rehabilitation Journal. 2018; 16:353-60. DOI
17. Brown TA. Confirmatory factor analysis for applied research. Guilford publications: New York ; 2015.
18. Tabachnick BG, Fidell LS, Ullman JB. Using multivariate statistics. Allyn &amp; Bacon/Pearson Education: Boston ; 2007.
19. Plichta SB, Kelvin EA, Munro BH. Munro’s statistical methods for health care research. Wolters Kluwer Health/Lippincott Williams &amp; Wilkins: London ; 2013.
20. Hooper D, Coughlan J, Mullen M. Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods. 2008; 6:53-60.
21. Schreiber JB, Nora A, Stage FK, Barlow EA, King J. Reporting structural equation modeling and confirmatory factor analysis results: A review. J Educ Res. 2006; 99:323-38. DOI
22. Polit DF, Yang F. Measurement and the measurement of change: a primer for the health professions. Wolters Kluwer Health: New York ; 2015.
23. Carter R, Lubinsky J. Rehabilitation research-e-book: Principles and applications. Elsevier Health Sciences: New York ; 2015.
24. Gillespie DF, Perron BE. Key Concepts in Measurement. Oxford University Press: Oxford ; 2015.
25. Diekema SM, Deitz J, Amundson SJ. Test-retest reliability of the evaluation tool of children’s handwriting-manuscript. Am J Occup Ther. 1998; 52:248-55. DOI | PubMed
26. McDowell I. Measuring health: a guide to rating scales and questionnaires. Oxford University Press: Oxford; 2006.
27. De Vet HC, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine: a practical guide. Cambridge University Press: Cambridge ; 2011.
28. Yong AG, Pearce S. A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutor Quant Methods Psychol. 2013; 9:79-94. DOI
29. Feder KP, Majnemer A. Handwriting development, competency, and intervention. Dev Med Child Neurol. 2007; 49:312-7. DOI | PubMed
30. Graham S, Boyer-Shick K, Tippets E. The validity of the handwriting scale from the test of written language. J Educ Res. 1989; 82:166-71. DOI
31. Hammerschmidt SL, Sudsawad P. Teachers’ survey on problems with handwriting: referral, evaluation, and outcomes. Am J Occup Ther. 2004; 58:185-92. DOI | PubMed
32. Ziviani J, Elkins J. An evaluation of handwriting performance. Educ Rev. 1984; 36:249-61. DOI
33. Daniel ME, Froude EH. Reliability of occupational therapist and teacher evaluations of the handwriting quality of grade 5 and 6 primary school children. Aust Occup Ther J. 1998; 45:48-58. DOI