How does the way in which a question is posed affect test-takers' responses ? An empirical investigation using a language comprehension test

The purpose of the present study was to examine if the way that questions are posed on a Japanese language reading test affects the test-takers' responses. The following variations of item presentation were used: (a) an item format in which test-takers chose a sentence (multiple choice or free response), (b) a presentation style in which blanks were the same throughout or varied in form, the number of blanks varied (many or few), and blanks were presented with or without an equal sign between them, and (c) a format that either did or did not have descriptive phrases before the blanks. Junior high school students (N= 703) were given 50 minutes to complete the test, and the proportion of correct answers and item discrimination were analyzed. The results were as follows: (a) Item formats involving choosing a sentence did not affect the proportion of correct answers or item discrimination, (b) Presenting blanks without an equal sign between them, and varying the presentation of blanks, resulted in an increase in the proportion of correct answers and item discrimination. When the presentation of blanks was varied, presenting blanks without an equal sign between them led to a higher proportion of correct answers. Higher discrimination was found when the blanks used the same form, when fewer blanks were used, and when blanks were presented without an equal sign between them. When complicated texts, such as multiple blanks, or blanks of the same form, were combined with blanks with an equal sign between them, a higher proportion of correct answers was obtained, (c) The data on inclusion/exclusion of descriptive phrases before the blanks showed an interaction with gender. The present study suggests the importance of considering the way in which questions are posed in tests.