Evaluating Usability in Technology-Enhanced Learning - 100 Ways Do It Right?

I am currently attending several paper presentation during the EC-TEL 2007. Many papers present new tools for authoring, connecting people, aggregating content etc. Most have evaluated their recently developed tools. That is the most interesting part, isn’t it? I have seen evaluating the efficiency, effectivity, usability etc. Especially in the domain of usability evaluation “standards” exist. At least everybody is stating that he is using “the standard”. Even if I thought that I know the standards for measuring usuability, I have to say that in technology-enhanced learning new instruments are said to be a standard instrument. Cool! So I spent some time in page turning the proceedings. My first impression was that usability is often evaluated by using self-developed measuring instruments. Why? Are TEL tools so different from other software? The reason for using standards measure instruments is to compare the usability of TEL tools. Using proprietary things makes it impossible to compare tools.
I am working at a software engineering institute, we mostly use the “technology acceptance model” also known as TAM or the Unified Theory of Acceptance and Use of Technology model also called UTAUT. These scales have been evolved over the years and work fine - at least for us. If I ask people ask the people whether they know TAM or UTAUT, I get the answer “well in fact - no”. One of the speakers uses “SUS” to evaluate the usability. SUS stands for quite and dirty scale for usability evaluation. I had a look at it: the webpage not available anymore. I only find a paper about it as PDF. The scale look ok - put creating to a scale value looks very suspicious because very different items were just summed up with some weights …

Could it be that usability evaluation is dependent on the context of the tools to be evaluated or is it just another case of “not knowing and looking at other domains”? One thing is for sure. If we continue to use different “standards” or self-developed instruments, we will never be able to compare things….(ok it is always a risk to compare your stuff with other stuff in the right way - you risk to loose your face ;-))

Leave a Reply