version 1.0

The Human-Computer Interaction Evaluation Ontology (HCIEO)

domain ontology from HCI-ON


1. Ontology Description

The Human-Computer Interaction Evaluation Ontology (HCIEO), a reference ontology that aims to provide a common conceptualization of HCI Evaluation and is a domain ontology of HCI-ON.

The purpose of the HCI Evaluation Ontology (HCIEO) is to establish an explicit and shared conceptualization of HCI evaluation by means of a human-centred design view, describing the main concepts involved in this context.

HCIEO covers relevant aspects of the HCI evaluation, such as the main involved artifacts, considered evaluation criteria, evaluated characteristics, involved agents and measures that can be used to evaluate an interactive computer system.


2. Related Ontologies

Networked ontologies used by HCIEO:


3. Ontology Models

Figure 1 presents the packages of the HCIEO Modularization.

Figure 1. HCIEO Modularization.


3.1. Central View

Figure 2 presents the conceptual model of the HCIEO Central View subontology.

Figure 2. HCIEO Central View conceptual model.

HCI Evaluation is a Complex Action and, like that, it is an intentional event. It is caused by the HCI Evaluator Intention that inheres in the HCI Evaluator. Being an Intention, HCI Evaluaton Intention has a HCI Evaluator Goal (Goal) as propositional content. HCI Evaluator is the role played by an individual or an organization (Stakeholder) that performs the HCI Evaluation.

An HCI Evaluation consists in the systematic determination of the extent to which an Interactive Computer System ’s (or its User Interface ) quality characteristics (HCI Quality Characteristic) meet the HCI Evaluation Criteria applied in the evaluation. HCI Evaluation Criteria, in turn, are conditions, or capacity needed (Requirement), used to evaluate Interactive Computer System ’s HCI Quality Characteristics and may be related to User Requirements.

HCI Quality Characteritics are qualities (Quality Universal) of an Interactive Computer System (manifested in its software or hardware constituents). Therefore, HCI Evaluation evaluates HCI Quality Characteristics of an Interactive Computer System or, more specifically, of its User Interface.

An HCI Evaluation follows HCI Evaluation Method (e.g., Heuristic Method, Usability Testing), which is a Procedure that describes the actions to be performed by the HCI Evaluator in an HCI Evaluation. An HCI Evaluation Method is chosen according to the HCI Evaluator Goal and can determine the HCI Evaluation Criteria to be applied in an HCI Evaluation. To perform an HCI Evaluation it may be necessary to use an HCI Evaluation Artifact (Sw Artifact), which is an object intentionally built to specify, among others, the HCI Evaluation Criteria to be applied in the evaluation (e.g., a checklist specifying the criteria to be applied).

In some HCI Evaluations it is necessary to evaluate the Interactive Computer System during the Human-Computer Interaction . In this case, the User (or user representatives) of the Interactive Computer System participates in the HCI Evaluation while, at the same time, participates (User Participation ), together with the Interactive Computer System (Interactive Computer System Participation ), in the Human-Computer Interaction . When the User participates in an HCI Evaluation, it is required that this User uses the Interactive Computer System in order to achieve a certain goal (User Goal ) (e.g., send an email). While the User interacts with the Interactive Computer System to achieve his/her goal (User Goal ), the HCI Evaluator performs the HCI Evaluation, following the HCI Evaluation Method and applying HCI Evaluation Criteria to evaluate HCI Quality Characteristics of the Interactive Computer System . This scenario involves some constraints, addressed by the following axioms:

A6.1. ∀ hcie: HCI Evaluation, ics: Interactive Computer System, icsp: Interactive Computer System Participation, hci: Human-Computer Interaction, user: User evaluates(hcie, ics) ∧ participatesIn(user, hcie) ∧ composedOf(hci, icsp) → ∃ up: User Participation participationOf(user, up) ∧ composedOf(hci, up)

A6.2. ∀ hcie: HCI Evaluation, hci: Human-Computer Interaction, hciqc: HCI Quality Characteristic, ics: Interactive Computer System, icsp: Interactive Computer System Participation evaluates(hcie, hci) ∧ evaluates(hcie,hciqc) ∧ has(ics, hciqc) ∧ composedOf(hci, icsp) ∧ participationOf(icsp, ics) → evaluates(hcie, ics)

Moreover, in an HCI Evaluation an HCI Evaluator can act as a User . Conversely, a User can act as an HCI Evaluator. Thus, if an evaluator performs an evaluation in which he/she acts as a user, then this evaluator is also the user who participates in the evaluation. On the other hand, if a user that participates in an evaluation performs that evaluation, then he/she is also an evaluator of that evaluation. Hence, the following axiom applies:

A6.3. ∀ e: HCI Evaluator, hcie: HCI Evaluation performs(e, hcie) ∧ participatesIn(e, hcie) → ∃ u: User participatesIn(u, hcie) ∧ (e = u)

A6.4. ∀ u: User, hcie: HCI Evaluation participatesIn(u, hcie) ∧ performs(u, hcie) → ∃ e: HCI Evaluator performs(e, hcie) ∧ (e = u)

After the HCI Evaluation is performed, the HCI Evaluation Report, a Document, is created, under responsibility of the HCI Evaluator, aiming to report the evaluation results and other relevant information, such as the applied HCI Evaluation Criteria.


3.2. Complementary View

Figure 3 presents the conceptual model of the HCIEO Complementary View subontology.

Figure 3. HCIEO Complementary View conceptual model.

There are HCI Evaluations in which it is necessary to perform measurements to determine the extent to which HCI Quality characteristics of an Interactive Computer System meet HCI Evaluation Criteria. This is addressed in the conceptual model shown in Figure 6.5, which depicts a view of HCIEO that reuses concepts from COM to represent evaluations that quantify HCI Quality Characteristics.

Besides being Objects (grounding in UFO), Interactive Computer System and User Interface are also Measurable Entities, i.e., entities that can be measured. In this sense, HCI Quality Characteristic is subtype of Measurable Element, i.e., a measurable property that characterizes Interactive Computer System (or a User Interface ). Measures can be used to quantify HCI Quality Characteristics. For example, usability (HCI Quality Characteristic) characterizesasystem(Interactive Computer System)oritsUI(User Interface)and can be quantified, among others, by means of the number of wrong clicks or touches (Measure). HCI Evaluation Criteria can refer to Measures used to quantify HCI Quality Characteristic. For example, the criteria “the number of wrong clicks or touches must be smaller than two”, used to evaluate usability, refers to the measure number of wrong clicks or touches, which, in turn, is used to quantify usability. In this context, the Measure referred by an HCI Evaluation Criteria in an HCI Evaluation that evaluates a HCI Quality Characteristic must be a Measure used to quantify that HCI Quality Characteristic. Thus:

A6.5. ∀ m: Measure, hciec: HCI Evaluation criteria, hcie: HCI Evaluation, hciqc: HCI Quality Characteristic evaluates(hcie, hciqc) ∧ applies(hcie, hciec) ∧ usedToEvaluate(hciec, hciqc) ∧ refersTo(hciec, m) → quantifies(m, hciqc)

A Measure (e.g., time spent to log in the system) can be expressed in Measure Unit (e.g., minute) and has a Scale partitioned according to the Measure Unit and composed of the values that can be associated to the Measure. Measurement consists in collecting Measured Values to a Measure (e.g., the measurement of the time to log in the system, resulting in the value 1 minute). In an HCI Evaluation, Measurements are performed to establish Measured Values to quantify HCI Quality Characteristics.

When a HCI Evaluation involves measurements, the HCI Evaluation Report must consider the Measured Values.


4. Concepts Definition

The following table shows the definitions for HCIEO concepts.

Concept Definition

HCI Evaluation

Intentional event (i.e., a Complex Action) caused by the HCI Evaluator Intention and that consists in the systematic determination of the extent to which an Interactive Computer System (or its User Interface) quality characteristics meet the HCI Evaluation criteria considered in the evaluation.
E.g.: evaluation of the usability of the checkout system of an e-commerce web site.
src.: adapted from (ISO/IEC 14598-5:1998; ISO/IEC 25066, 2016; ISO/TS 18152, 2010; ISO/IEC TR 25060, 2010; ISO/TR 16982, 2002; ISO 9241-220, 2019).

HCI Evaluation Artifact

Object intentionally made to specify evaluation criteria in the context of an evaluation.
E.g.: a checklist used during an evaluation to specify the criteria to be applied.

HCI Evaluation Criteria

Conditions, or capacity needed (Requirement), used to evaluate interactive computer system’s quality characteristics. It can be determined by the HCI evaluation method of a certain HCI evaluation and may be related to user requirements.
E.g.: total time the user spent on the first attempt to execute the task versus the time on the second attempt (the purpose of this data is to understand if the user can achieve a learning curve and on the second attempt to achieve the goal more quickly). In this case, the quality characteristic is usability, more specifically its learnability component.
src.: adapted from (ISO/IEC TR 25060, 2010; ISO 9241-220, 2019).

HCI Evaluation Method

A Procedure describing the actions to be performed by the HCI Evaluator to perform a HCI Evaluation.
E.g.: heuristic evaluation, cognitive walkthroughs, standards inspection, pluralistic walkthroughs, usability test, consistency inspections, among others.
src.: adapted from (ISO/IEC 14598-5, 1998; ISO/TR 16982, 2002).

HCI Evaluation Report

Document that presents evaluation results and other relevant information such as the considered HCI Evaluation Criteria.
E.g.: Usability test report.
src.: adapted from (ISO/IEC 14598-5, 1998; ISO/IEC TR 25060, 2010).

HCI Evaluator

HCI Evaluator & Individual or organization (Stakeholder) that performs an HCI Evaluation.
E.g.: usability specialists, developers.
src.: adapted from (ISO/TR 25060, 2010; ISO/IEC 25040, 2011).

HCI Evaluator Goal

The evaluation intended outcome.
E.g.: Make the web site login UI more usable and understandable.

HCI Evaluator Intention

Intention that inheres in a HCI Evaluator, causing him to perform an HCI Evaluation.
E.g.: the intention to evaluate an interactive computer system to find usability problems.

HCI Quality Characteristic

Characteristic expected in an Interactive Computer System (in software or hardware constituents), e.g., usability (software), screen size of a smart watch (hardware), among others. It can refer to the whole Interactive Computer System or part of it (e.g., its User Interface an Input Equipment).
E.g.: usability of a smart watch, communicability of an UI login.
src.: adapted from (ISO 9241-11, 2018).

HCIEO concepts definition.

5. Detailed Concepts

Human-Computer Interaction Evaluation Ontology (HCIEO) detailed concepts.


HCIEO::HCI Evaluation


HCI Evaluation


Specializes:

UFO::Complex Action

Definition:

Intentional event (i.e., a Complex Action) caused by the HCI Evaluator Intention and that consists in the systematic determination of the extent to which an Interactive Computer System (or its User Interface) quality characteristics meet the HCI Evaluation criteria considered in the evaluation.
Example: evaluation of the usability of the checkout system of an e-commerce web site.
Source: adapted from (ISO/IEC 14598-5:1998; ISO/IEC 25066, 2016; ISO/TS 18152, 2010; ISO/IEC TR 25060, 2010; ISO/TR 16982, 2002; ISO 9241-220, 2019).

Relations:

HCI Evaluation (0..*) follows (1..*) HCI Evaluation Method
HCI Evaluation (1..1) creates (1..1) HCI Evaluation Report
HCI Evaluation (0..*) caused by (1..1) HCI Evaluator Intention
HCI Evaluation (0..*) evaluates (1..*) HCI Quality Characteristic
HCI Evaluation (0..*) evaluates (1..*) Interactive Computer System Participation
HCI Evaluation (1..1) uses (0..*) HCI Evaluation Artifact
HCI Evaluation (0..*) applies (1..*) HCI Evaluation Criteria
HCI Evaluation (0..*) evaluates (1..*) Human-Computer Interaction
HCI Evaluation (0..*) evaluates (1..1) Interactive Computer System
HCI Evaluation (0..*) evaluates (1..1) User Interface
HCI Evaluator (1..*) performs (1..*) HCI Evaluation
User (0..*) participates in (0..*) HCI Evaluation


HCIEO::HCI Evaluation Artifact


HCI Evaluation Artifact


Specializes:

SysSwO::Software Artifact

Definition:

Object intentionally made to specify evaluation criteria in the context of an evaluation.
Example: a checklist used during an evaluation to specify the criteria to be applied.

Relations:

HCI Evaluation Artifact (0..*) specifies (1..*) HCI Evaluation Criteria
HCI Evaluation (1..1) uses (0..*) HCI Evaluation Artifact


HCIEO::HCI Evaluation Criteria


HCI Evaluation Criteria


Specializes:

RSRO::Requirement

Definition:

Conditions, or capacity needed (Requirement), used to evaluate interactive computer system’s quality characteristics. It can be determined by the HCI evaluation method of a certain HCI evaluation and may be related to user requirements.
Example: total time the user spent on the first attempt to execute the task versus the time on the second attempt (the purpose of this data is to understand if the user can achieve a learning curve and on the second attempt to achieve the goal more quickly). In this case, the quality characteristic is usability, more specifically its learnability component.
Source: adapted from (ISO/IEC TR 25060, 2010; ISO 9241-220, 2019).

Relations:

HCI Evaluation Criteria (0..*) used to evaluate (1..*) HCI Quality Characteristic
HCI Evaluation Criteria specified according to HCI Evaluator Goal
HCI Evaluation Criteria (0..*) refers to (0..*) Measure
HCI Evaluation Criteria (0..*) refers to (0..*) User Requirement
HCI Evaluation (0..*) applies (1..*) HCI Evaluation Criteria
HCI Evaluator Goal (0..*) helps to determine (1..*) HCI Evaluation Criteria
HCI Evaluation Method (0..*) can determine (0..*) HCI Evaluation Criteria
HCI Evaluation Report (1..*) reports (1..*) HCI Evaluation Criteria
HCI Evaluation Artifact (0..*) specifies (1..*) HCI Evaluation Criteria


HCIEO::HCI Evaluation Method


HCI Evaluation Method


Specializes:

SPO::Procedure

Definition:

A Procedure describing the actions to be performed by the HCI Evaluator to perform a HCI Evaluation.
Example: heuristic evaluation, cognitive walkthroughs, standards inspection, pluralistic walkthroughs, usability test, consistency inspections, among others.
Source: adapted from (ISO/IEC 14598-5, 1998; ISO/TR 16982, 2002).

Relations:

HCI Evaluation Method (0..*) can determine (0..*) HCI Evaluation Criteria
HCI Evaluation Method (1..*) chosen according to (0..*) HCI Evaluator Goal
HCI Evaluation (0..*) follows (1..*) HCI Evaluation Method
HCI Evaluator Goal (1..1) helps to choose (1..*) HCI Evaluation Method
HCI Evaluator (1..*) choose (1..*) HCI Evaluation Method


HCIEO::HCI Evaluation Report


HCI Evaluation Report


Specializes:

SysSwO::Document

Definition:

Document that presents evaluation results and other relevant information such as the considered HCI Evaluation Criteria.
Example: Usability test report.
Source: adapted from (ISO/IEC 14598-5, 1998; ISO/IEC TR 25060, 2010).

Relations:

HCI Evaluation Report (1..*) reports (1..*) HCI Evaluation Criteria
HCI Evaluation Report (0..*) reports (0..*) Measured Value
HCI Evaluation (1..1) creates (1..1) HCI Evaluation Report
HCI Evaluator (1..*) / responsible for (1..*) HCI Evaluation Report


HCIEO::HCI Evaluator


HCI Evaluator


Specializes:

SPO::Stakeholder

Definition:

HCI Evaluator & Individual or organization (Stakeholder) that performs an HCI Evaluation.
Example: usability specialists, developers.
Source: adapted from (ISO/TR 25060, 2010; ISO/IEC 25040, 2011).

Relations:

HCI Evaluator (1..*) performs (1..*) HCI Evaluation
HCI Evaluator (1..*) / responsible for (1..*) HCI Evaluation Report
HCI Evaluator (1..*) choose (1..*) HCI Evaluation Method
HCI Evaluator Intention (1..*) inheres in (1..1) HCI Evaluator


HCIEO::HCI Evaluator Goal


HCI Evaluator Goal


Specializes:

UFO::Goal

Definition:

The evaluation intended outcome.
Example: Make the web site login UI more usable and understandable.

Relations:

HCI Evaluator Goal (1..1) propositional content of (1..*) HCI Evaluator Intention
HCI Evaluator Goal (0..*) helps to determine (1..*) HCI Evaluation Criteria
HCI Evaluator Goal (1..1) helps to choose (1..*) HCI Evaluation Method
HCI Evaluation Method (1..*) chosen according to (0..*) HCI Evaluator Goal
HCI Evaluation Criteria specified according to HCI Evaluator Goal


HCIEO::HCI Evaluator Intention


HCI Evaluator Intention


Specializes:

UFO::Intention

Definition:

Intention that inheres in a HCI Evaluator, causing him to perform an HCI Evaluation.
Example: the intention to evaluate an interactive computer system to find usability problems.

Relations:

HCI Evaluator Intention (1..*) inheres in (1..1) HCI Evaluator
HCI Evaluation (0..*) caused by (1..1) HCI Evaluator Intention
HCI Evaluator Goal (1..1) propositional content of (1..*) HCI Evaluator Intention


HCIEO::HCI Quality Characteristic


HCI Quality Characteristic


Specializes:

UFO::Quality

UFO::Measurable Quality Universal

COM::Measurable Element

Definition:

Characteristic expected in an Interactive Computer System (in software or hardware constituents), e.g., usability (software), screen size of a smart watch (hardware), among others. It can refer to the whole Interactive Computer System or part of it (e.g., its User Interface an Input Equipment).
Example: usability of a smart watch, communicability of an UI login.
Source: adapted from (ISO 9241-11, 2018).

Relations:

HCI Quality Characteristic (1..*) inheres in (1..*) Human-Computer Interaction
HCI Evaluation (0..*) evaluates (1..*) HCI Quality Characteristic
User Interface (1..*) /has (0..*) HCI Quality Characteristic
Interactive Computer System (1..*) has (0..*) HCI Quality Characteristic
HCI Evaluation Criteria (0..*) used to evaluate (1..*) HCI Quality Characteristic