The TeSLA system will be tested in successive pilots of increasing levels of complexity and scale. As an important preparatory step universities involved in the TeSLA pilots have analysed previous pilot experiences in similar educational scenarios and documented this in Deliverable 7.1. A report full of noteworthy considerations and recommendations regarding design, methodologies, and strategies used in pilot studies.
Reading the report it occurred that the three main quality indicators often associated with assessments – transparency, validity, and reliability – apply to our pilots as well.
In piloting the TeSLA system transparency is of the utmost importance: the technologies used involve legal and ethical issues in an already highly sensitive context of personal assessment. If we are to motivate students and teachers and ensure their participation in the pilots, transparency about the technologies used – what they do, when and how to use them – is paramount. The report stresses the importance of clearly communicating expectations towards all stakeholders: students, teachers, technical & support staff, and senior management. For this purpose an initial pilot is envisioned, which will focus not so much on technology yet, but rather on establishing clear communication about the pilots and technologies e.g. through protocols and guidelines.
Regarding validity (the extent to which the pilot studies measure what they are supposed to measure) a lesson drawn in from previous experiences is the need for focus. A possible risk here is that the focus will shift from technology use to technical issues. The focus should be on application of the technology in an educational context, i.e. on usability. The concept of usability [1] covers four aspects: effectiveness, efficiency, attractiveness and accessibility. We can only obtain valid data on the usability of the TeSLA system if the technology itself is sufficiently tried and tested, otherwise we might end up with poor results not knowing whether they should be attributed to poor technical performance or poor usability. The approach of incrementally adding functionality in the medium test-bed pilots should help to avoid this. Another challenge concerning validity of the TeSLA pilots is related to what is called the ecological validity, i.e. the question whether we will be able to conduct the pilots in a real-world setting. Especially regarding high-stake, summative assessments, we can expect stakeholders to be reluctant if not outright opposed to any experimentation. Again clear communication towards teachers and students about risks and possible contingency planning needed, will be of paramount importance.
Reliability, finally, involves the methods and instruments used to collect data during the pilots, and whether or not they yield consistent results. The partners involved in the pilots all have extensive experience working with a variety of both qualitative and quantitative research methods and instruments. A mixed methods approach is likely to be adopted in order to capture data at various levels of detail and depth. In order to ensure comparisons can be made and subsequent conclusions drawn across various assessment scenarios and scenario implementations, it will be helpful to use a validated scale like the System Usability Scale [2]. This well documented instrument offers the additional benefit of easily accessible benchmark data, enabling us to not only compare the usability of the TeSLA system across various settings, but to compare it with the usability of other related software systems as well.
Jose Janssen
Assistant Professor at the Open University and Educational Manager at TeSLA Project
References:
[1] ISO (2015). Ergonomics of human-system interaction – Part 11: Usability: Definitions and concepts. International Organisation for Standardisation: ISO/DIS 9241-11. https://www.iso.org/obp/ui/#iso:std:63500:en
[2] Tullis, T., & Albert, W. (2008). Measuring the User Experience. Interactive Technologies Series. Burlington, MA: Morgan Kaufmann.
FUNDED BY THE EUROPEAN UNION
TeSLA is not responsible for any contents linked or referred to from these pages. It does not associate or identify itself with the content of third parties to which it refers via a link. Furthermore TESLA is not liable for any postings or messages published by users of discussion boards, guest books or mailing lists provided on its page. We have no control over the nature, content and availability of any links that may appear on our site. The inclusion of any links does not necessarily imply a recommendation or endorse the views expressed within them.
TeSLA is coordinated by Universitat Oberta de Catalunya (UOC) and funded by the European Commission’s Horizon 2020 ICT Programme. This website reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.