Multiple-Choice Assessments - our Assessing Adult Learning Assignment!

Hi group 2!

On doing a quick search of the articles about multiple choice questioning (MCQ) assessment I’ve found some (mainly US – based) research about HE issues, but also a lot of papers about either the self-management of patients’ chronic conditions or MCQ methods to measure the competence of their health professionals. It seems that there are many published papers about the biases involved in MCQ papers themselves (learning-styles/gender/ethnicity etc) or biases found in the correct answers (apparently these are frequently the longest ones – see below abstract)! Here are two web sites with a bit of info that you may find of use:

http://scholar.lib.vt.edu/ejournals/JTE/v6n1/haynie.jte-v6n1.html

http://scholar.lib.vt.edu/ejournals/JVME/V20-3/wilson.html


I will dig a bit deeper to find out about their use in the teaching of assessment to give us a clue where to start. Meanwhile have a look at these….

Response Biases in Multiple-Choice Test Item Files
Thomas L. Mentzer
University of New Haven
Thirty-five files of multiple-choice test items were examined for evidence of biases in the correct answers. Of seven types of biases, the most frequently occurring (24 cases) was the "All of the above" bias in which that answer was correct significantly more than 25% of the time. Another common bias (17 cases) was a tendency for the longest answer to be correct too frequently. Ten cases were also found in which answers of the form "Both A and B" were correct too frequently. Finally, the ABCD bias, in which the probability of each of the four response choices being correct differs from .25, was present in seven files. Three other biases occurred only a few times. It was concluded that authors should take precautions to prevent biases in future item files.
First web-based GCSE launched (TES 09/10/06)PA NewsPublished: 09 October 2006
The first GCSE to be entirely "e-assessed" was launched by an exam board today. Pupils will sit computer-based tests involving a mixture of multiple choice and short answers without any need for pens or pencils. Coursework for the new Environmental and Land-Based Science GCSE will also be submitted electronically. Marking for the multiple choice section of exams will be done by computer but other answers and coursework will be assessed by moderators. The GCSE has been developed by the OCR exams board, which said it would "put an end to endless reams of paper for both students and teachers". Storing work online eliminated the risk of mislaying papers and presenting work electronically allowed students to "take a much slicker and more professional approach to their work". The board said it would "transform the future of school examinations". Pupils are starting the GCSEs this term and will take their first exams in June next year before taking their final tests in 2008

"The dumbed-down GCSE exam with no need for writing"
TES: 12 June 2006
A new 'dumbing-down' row has erupted over Government plans to introduce GCSEs assessed entirely through multiple-choice questions. From November pupils across the country will be able to score up to 75 per cent in a science GCSE by choosing A, B, C or D options. The exam, run by the AQA and Edexcel boards, is the first major GCSE to be mainly multiple choice. Edexcel’s website says the exams give pupils "more chances to succeed". The row, first reported in this week’s TES, will fuel controversy over GCSEs and A-levels. Writing in The TES, Anthony Seldon, master of Wellington college, in Berkshire, says that confidence in the system is "draining quicker than credibility in George W Bush’s stance on Iraq". Meanwhile, Derek Bell, chief executive of the Association for Science Education said: "This seems to be an oversimplistic way of assessing what is a very complex learning process." The Conservatives were naturally quick to jab a stick at the Government’s plans. Nick Gibb, the Shadow Schools Minister, argued that an examination system is required that "genuinely tests the knowledge and skills children acquire at school" and not presumably not a dumbed-down system that simply "enables children to be trained for the exam itself", reports the Times.

"Pick A, B or C for a GCSE"
Warwick Mansell TES 09 June 2006
Hundreds of thousands of pupils will soon be able to pass GCSEs assessed entirely through multiple-choice questions and coursework. From November, pupils will be able to score up to 75 per cent in a science GCSE by choosing from A, B, C or D options. They will sit six multiple-choice papers of either 20 or 30 minutes, each of which is offered with up to five re-sits and is marked by computer. The exam, run by the AQA and Edexcel boards, is the first major GCSE to be mainly multiple-choice. The rest of the qualification comprises new-style coursework which allows pupils up to six attempts to get a good mark. Edexcel’s website says the exams give pupils “more chances to succeed” and that they can be tested “any time, allowing them to be tested on material when it’s fresh, and can take multiple tests”. Teachers will award up to 10 per cent of the marks for the Edexcel exam, and the board will not usually check their decision. The exams, which are also the first mainstream GCSEs to allow pupils to take some modules by computer, are provoking intense controversy. Derek Bell, chief executive of the Association for Science Education, said: “This seems to be an oversimplistic way of assessing what is a very complex learning process.” Jonathan Osborne, professor of science education at King’s college, London, said : “How is this going to assess pupils’ ability to express themselves in scientific language, a major aspect of science?” Critics say the new exams have been designed to be simple to mark, and to make it easy for pupils to get top grades. But the boards say that multiple-choice, or “objective tests”, can be a good way to check pupils’ understanding.
The development will fuel controversy over GCSEs and A-levels. Anthony Seldon, master of Wellington college, in Berkshire says confidence in the system is “draining quicker than credibility in George W Bush’s stance on Iraq”. Mr Seldon is organising a conference on June 27 in which heads of private schools will debate alternatives to mainstream exams. The new core science GCSEs are part of a set of courses designed to make the subject more exciting. Areas of study include genetic engineering, global warming and nanotechnology. Most pupils will take the exam alongside a second GCSE in additional science which is being assessed more traditionally. For coursework, pupils will complete up to six unassessed investigations. Their teachers will then set tests designed by the board on each project that will be conducted in exam conditions. The best mark will count towards their grade. An AQA spokeswoman said: “The variety of assessment processes we use reflects the diversity of skills being tested across a wide range of subjects. All our specifications have been subject to Qualifications and Curriculum Authority accreditation procedures.” A spokeswoman for the Edexcel board said its new courses were also accredited by the QCA and designed by science teachers. A QCA spokesman said the new specifications had yet to receive final accreditation, because the regulator believed that example questions suggested by the boards were not sufficiently “rigorous” but the concept of multiple-choice exams was something the QCA was happy with......

more later!!
happy reading!