Project 1: Evaluative Instrument Critique
Project 1 Evaluative Instrument Critique
Possible points (maximum): 30 points
Due: Sept. 23
Project Description
In this project, you will critically analyze and enhance an evaluative instrument that is designed to assess a learning technology-related training program or event. The selected instrument may be paper-based or electronic and can include formats such as a rubric, interview guide, focus group protocol, observation checklist, questionnaire, or survey. The primary objective of this project is to evaluate the effectiveness and appropriateness of the instrument for its intended evaluative purpose and to provide actionable recommendations for improvement.
Select an evaluative instrument (paper or electronic-based). It can be an interview guide, a focus group protocol, an observation checklist, a questionnaire, a survey, etc. The instrument must serve an evaluative (not research) purpose and should measure aspects of a learning technology-related training program or event.
Steps to Complete the Project:
1. Selection of the Evaluative Instrument
-Choose an evaluative instrument that is specifically designed to measure aspects of a learning technology-related training program or event. Ensure that the instrument is intended for evaluation (e.g., assessing the effectiveness, usability, or outcomes of a program) rather than for pure research purposes (e.g., exploring new theories or generating new knowledge).
2. Instrument Overview and Context
-Provide a brief summary of the selected instrument, including its intended purpose, target audience, and context of use. Clearly explain why this instrument is evaluative and not research-focused. Include a full reference for the instrument, detailing the authors, designers, developers, year of publication, title, publisher, or sponsoring organization.
3. Critical Analysis of the Instrument
-Purpose and Appropriateness: Assess the appropriateness of the instrument in relation to its stated evaluative purpose. Does it effectively measure the intended aspects of the training program or event?
-Strengths and Weaknesses: Identify the strengths and weaknesses of the instrument, particularly in terms of its data collection method. Consider factors such as validity, reliability, and practicality (refer to Westat, 2010, pages 58-73, for guidance).
-Item Quality: Evaluate the quality of the items within the instrument. Are the items clear, concise, and relevant? Do they avoid leading questions and potential biases? How well do they facilitate accurate and meaningful responses?
-Bias and Objectivity: Analyze the instrument for any signs of bias or leading questions. Assess whether the instrument allows for objective responses or if it tends to steer respondents toward a particular conclusion.
4. Recommendations for Enhancement
-Based on your critique, provide specific recommendations for revising and enhancing the instrument. Your recommendations should be actionable and aimed at improving the instrument’s effectiveness, accuracy, and usability.
-Offer a rationale for each recommendation, explaining how it will address the identified weaknesses or enhance the instrument’s strengths.
5. Report Presentation
- Prepare a professional critique report that clearly communicates your findings and recommendations to a “client audience” (i.e., someone who has hired you to provide an expert evaluation of the instrument).
- Include excerpts or reproductions of the instrument with annotations or suggestions for revisions where relevant.
- Ensure that the report is well-organized, clearly written, edited, and proofread to a professional standard.
6. Submission Format
- Submit your critique report as a PDF document.
- Length: 1,500 to 2,000 words (excluding references and appendices).
Deliverables
Critique report: 1,500 to 2,000 words critique report – references and appendices do not count towards the word count.
Points: 30
Submission: Submit your critique report by Monday, Sept. 23 at 11: 59 p.m.
Evaluation Criteria for Critique Report
Your report will be evaluated based on:
- Inclusion of:
- Full reference of the evaluative instrument used for critique (authors, designers, developers, year of publication, title, publisher, or sponsoring organization) (3 points)
- A brief summary or blurb of the instrument that is the object of the critique. Please explain why this is an evaluative and not a research instrument (3 points)
- Discussion of at least three criteria:
- Appropriateness of the instrument to the purpose of the evaluation stated (4 points)
- Strengths and weaknesses of the instrument in terms of a data collection method (4 points)
- Quality of the items created (4 points)
- Depth of the rationale presented for critique (4 points)
- Relevance and specificity of recommendations (6 points)
- Clarity of the report for a “client audience” and quality of the writing (2 points)
Westat, J.F. et al. (2010). The 2010 User-Friendly Handbook for Project Evaluation. Division of Research and Learning in Formal and Informal Settings, Directorate for Education and Human Resources. Arlington, VA: National Science Foundation. FREE copy at: https://www.purdue.edu/research/docs/pdf/2010NSFuser-friendlyhandbookforprojectevaluation.pdf
Example Projects
See some examples:
- Project Report Download Project Report
- Example Instrument: Evaluative Instrument BlackBoard Rubric Download Evaluative Instrument BlackBoard Rubric
- Project report and instrument 1 Download Project report and instrument 1
- Project report and example instrument 2 Download Project report and example instrument 2
- Project report example Download Project report example