TUI Vs. GUI Comparative Methods

TUIs vs. GUIs: comparing the learning potential with preschoolers.

Sylla, C., P. Branco, et al. (2012). “TUIs vs. GUIs: comparing the learning potential with preschoolers.”Personal and Ubiquitous Computing 16(4): 421-432.

This is a project where the main intention was to compare the learning potential that TUIs and GUIs have. This project intended to teach children about oral hygiene. The activity was introduced as germs that crawled on to the surface of a tooth. The user’s challenge was to brush away those germs. This same activity was used on both interfaces.


Subject group:
Two groups of children aged 4 to 5 years.
Group A 18 children and Group B 23 children. Both groups were from different schools and had no contact with each other. The children played with the interfaces individually meanwhile others were giving advice. 40 minutes was allocated for the TUI as 30 for the GUI.

Three different methodologies were used to assess how the interface changed children’s attitude toward tooth brushing.

1. Children’s attitude before and after being exposed to the interfaces by using a questionnaire sent to parents.

Questions given to the parents:
• Motivation of their children for tooth brushing
• Children’s opposition to tooth brushing
• Children’s notion of the importance of tooth brushing
• Children’s knowledge of the consequences of bad oral hygiene

The first approach to the interfaces was one week after receiving the answers from the questionnaires. The results from the first questionnaire showed no difference between both groups.

2. By using drawings, the researchers were capable of measuring the level of involvement that the children had with the interface.

The use of single image to communicate what they were capable to remember was a good approach since they describe all the elements that they managed to retain or the elements that most impressed them. This makes it easier to understand the impact the interface had on them. It is important to mention that there was a gap of five months in between the exchange of groups from the GUI to the TUI and viceversa.

Figure 1. Illustration of children who used the GUI

When analysing the images, realistic likeness is not relevant. Dealing with issues like proportion or extra realism may interfere with their conception of the object intended to be represented. At age 4, children are still at the preschematic stage. They try to reproduce objects that relate to their environment.

Figure 2.  Illustration in self portrait of the users that interacted with the TUI

3. The children were interviewed about their likes and dislikes of the interfaces.


According to their results, they showed that TUIs have big potential to promote learning. They are capable of providing long-lasting involvement making children to have a better engagement.

TUIs provide a long-lasting engagement that promotes learning, as presented on the left image in Figure 2. The image shows the tooth before and after the germs were removed indicating the elements and events that they remembered in their interactive engagement. On the other hand, they argue that GUIs lack the richness of human senses utilised in the real world limiting this way their communication channels.

The use of animation and tangible tools and the novelty effect [EC1] improve the engagement that children have. Children in the experiment have been previously exposed to computers. In this case, the TUI brought a new experience. They argue that novelty is not necessarily a problem central to this case but rather a characteristic of the TUI. I will argue that the novelty of the TUI may benefit the engagement from children over the GUI.

Papert has also noted through the constructionist approach the importance that physical objects have on the construction of knowledge. Also, Maria Montessori developed a method where children learn by playing with tractable objects. Further on Froebel-inspired Manipulatives combined the Montessori method of learning abstract concepts with the possibility of modelling of objects of the real world. For this reason groups including Lifelong Kindergarten at MIT Media Lab or the MIT Tangible Media Group have been involved in the use of TUIs to assist learning.


Storytelling through drawings: evaluating tangible interfaces for children

Sylla, Cristina, Pedro Branco, Clara Coutinho, and Maria Eduarda Coquet. 2009. Storytelling through drawings: evaluating tangible interfaces for children. In Proceedings of the 27th international conference extended abstracts on Human factors in computing systems. Boston, MA, USA: ACM.

This paper is based on the paper from the project previously presented. As part of one of their three methodologies used, they argue that assessing through drawings can be utilised. Younger children may have issues verbalising their thoughts, especially for children who cannot read. Methods such as the Draw-a-Person test (Quantitative Scoring System) are widely used to assess children’s cognitive development.

There were two groups of 4 year old children, 18 in group A and 23 in group B. First, group A tried the TUI and group B the GUI with the settings mentioned on the previous paper. Each group was sent to a different room where they were asked to draw their experience. In this exercise process, they were not given any questions or suggestions.

To evaluate the drawings they arranged drawn elements which were common to both interfaces into one group and another group for the remaining ones. Each of those elements was scored a point. The researchers used a “non-parametric Mann-Whitney U test” for independent groups.

There were some issues with interpretation of some of the drawings. To prevent false interpretation, the children were asked to explain what they drawn and the research team annotated the illustration.


By analysing the drawings, the researchers found that there were many elements that were common for both interfaces, group A (TUI) scored 2 points against 2,6 from group B. Furthermore, with the less common elements, group A scored 4,9 against 3 from group B. Only 9 out of the 23 children actually drew the ‘other elements’ while in group A 17 out of 18 were capable of remembering the ‘other elements’. When the groups changed from the TUI to the GUI later on in the experiment, the results were very similar. Children were capable of remembering more information when working with the TUI.


A Comparative Content Analysis of Student Interaction in Synchronous and Asynchronous Learning Networks

Chou, C. C. 2002. “A comparative content analysis of student interaction in synchronous and asynchronous learning networks.” Proceedings of the 35th Annual Hawaii International Conference on System Sciences. doi: 10.1109/hicss.2002.994093.

This project does not relate directly to TUIs but to analysing and comparing to different learning methods through synchronous and asynchronous leaning networks. Asynchronous learning is based on student-centred approach. The student is capable of choosing his learning activities when is most convenient for him. Most of the asynchronous learning is distributed by technology such as e-courses, wikis, and multimedia.

When analysing learning outcomes or working with interactive teaching, it is necessary to implement theoretical grounding of instructional design. In the same manner, I will argue that TUIs, the same as many GUIs are asynchronous learning methods where these methods for analysis can be implemented.

The Interaction Analysis Model was used to analyse the process of social construction of knowledge. The model is intended to analyse the social construction of knowledge. It is separated through different stages as an attempt to clarify how learners achieve a higher level of critical thinking. These steps are [1] comparison and share of information, [2] finding of inconsistencies, [3] negotiation / co-construction of knowledge, [4] testing and modification, and [5] agreement/application.

Method (Activities)

The course designed for testing was designed for an undergraduate course titled “Theories and Applications of Computer-Mediated Communication Systems” for the University of Hawaii. It’s main objective is to enrich understanding of CMCSs. For this these principles were implemented:

• Principle 1. Learner-centred instructional design: focused on cognitive, meta-cognitive, motivational, social and individual differences.

• Principle 2. Constructivist activities

• Principle 3. Small group cooperative learning.

The content was presented through various text-based, audio-video conferencing, and enhanced virtual systems. Three-member groups were selected by turns to moderate the seminar each week.

Method (Research):

This project used content analysis to examine users interaction with a distance learning course. There were three different contexts of interaction: learner-content, learner-instructor and learner-learner. The data was collected from weekly computer conferences held on WebCT (Blackboard Learning System) and chat rooms.

The coding scheme was based on Bales Interaction Process Analysis (IPA). The researchers focused on two areas: category 1-3, 10-12 from the socio-emotional section and category 4-9 from the task areas.

Figure 3. Bales IPA table

The transcripts from both learning systems were analysed separately.
A total of 4,977 sentences were coded. 2,519 were asynchronous discussions and 2,458 belonged to synchronous seminars. Two coders were used for the analysis using a software called NUD*IST, a “qualitative data analysis software”. The data was later on analysed used SPSS by IBM, a statistical analysis software.


  • On the beginning there were more issues with exchanges of personal information and frustration with the CMS system. As a result there was less SE-oriented interaction.
  • Later in the semester, students were familiar with the online tools. They were asking more task-oriented questions.
  • In synchronous communication mode, there were more interactions in showing support and personal information exchanges.
  • The communications in asynchronous mode were mostly one-way.

Moderators play a critical role in online seminars. There are issues with time management, moderators haste into conclusions due to insufficient time.
It is important to study and understand interaction factors to enhance the progress of learners, teachers and institutions.

TOK – a Tangible Interface for Storytelling

The researcher has been working with the use of storytelling for assessing children’s knowledge. She has argued that this method can be used also to asses the effectiveness of interfaces. She has especially focused in the comparison in between TUI and GUI.

This project is about a platform for preschool children to create their own stories. It uses 15 rectangular cards which are meant to be placed on slots. These cards react to create animations that are rendered on a screen. These images represent the set of actions that can be visualised on the storytelling rendering tool.

Figure 4. Conceptual render of the interface

Children can experiment generating different type of sequences. Once the desired card sequence is generated, the interface then proceeds to generate the animation. This allows the children to produce a story and to generate other stories through experimentation.

Children can explore and learn about things that surround them by creating stories about them. TOK provides an interface for storytelling.


25 preschool children aged five. They worked in small groups of 3.

  1. 1.     Designing the story

For 2 days where they were asked to produce any kind of story. This exercise was developed to understand what sort of stories children wanted to create. On those two sessions, children came up with characters such as: men, girl, boy, doctor, rabbit, piggy, grandma and cousin among others. They also came up with some scenery and some ‘nature’ such as: flower, trees and apples.

  1. 2.     Drawing the cards

After knowing what sort of stories the children wanted and what sort of characters, the team created three sets of picture cards: characters, places and actions. This is when they started using rectangular cards keeping in mind the possibilities of different shape usage.

  1. 3.     Testing the paper prototype

The Children were asked to create a story using the paper prototype. The cards were scattered on a table and they were free to choose any card and place them on an A4 cardboard to create any kind of story.


Children wanted to create more than one story once they had access to the cards. The generated stories with the paper prototype were different to the ones originally created on the design story process. The content on the cards was very clear for the children.

There were 30 stories in total. The next table shows the differences of how children used and placed the cards to create their story.

Begin top to bottom

Begin left to right

Follows the cards

Total stories















The analysis involved the behaviour of children and their process of placing the cards used for the story.  First comparing a process where they can freely create a story from scratch to one partially constrained by the cards provided. This study was used to understand children’s behaviours when using different type of storytelling systems.

The evidence showed that children were capable and motivated to create more stories when provided with the cards. The children used the cards to create stories and were keen to listen to the other children’s stories.

The proposed TUI prototype for storytelling will contain 15 slots to place cards.


GUI vs. TUI: Engagement for Children with No Prior Computing Experience

L. K. Cheng, C. S. Der, M. S. Sidhu and R. Omar, GUI vs. TUI: Engagement for Children with No Prior Computing Experience. Electronic Journal of Computer Science and Information Technology (eJCSIT), Vol. 3, No. 1, 2011

There has been several methods and previous research to compare TUI and GUI. When working with children, the researchers focus on adapting these methods into a more accurate approach. Many of the methods applied for comparing TUI and GUI are: Efficiency, learnability, fun, spatial cognition, controllability, usefulness, feedback and ease of use. Satisfaction and engagement can also be used in the analysis. When dealing with children’s satisfaction, which is a ‘grown-up word’ can be translated into fun, for example:

Enjoyment or fun


The fun that is attached to an event, and the fun as it is affected by the prior expectations of the user


The cognitive effort and deep processing of new information


Likehood to remember things that we have enjoyed.

Their research interests aimed to understand the satisfaction/engagement results when working with non-computer exposed children. This should become an important factor when measuring user’s performance. They argue that the ‘wow’ factor will be similar when exposed to the first time to a GUI or a TUI.

Project Design

This is a project focused on comparing TUI and GUI. They developed content for a Malaysian school involving children of 8 years of age. The teaching content was developed for a topic called “3D Shapes”. The researchers chosen this topic since they argue it will be a good initial exposure to 3D shapes. Both interfaces were design to interact with 3D shapes. There were some basic interactions with the solids such as: Rotate, Wireframe, Extract, Say It and Reset.

Figure 5. GUI vs TUI


Children samples were tested in pairs. All samples had prior experience with computing data. They followed an extensive selection process:

For time-registration the researchers could not implement any ‘direct triggering’ automation method to the project. For this, they used the facilitator and a digital chronograph. For the ‘Engagement’ section, the facilitator asked the child if he wanted to keep on using the interface. If he decided to keep on ‘playing’ with it, then the facilitator would start the clock until that child finishes playing. He will just become and observer and step back 10 metres away from the workspace.


The children’s engagement was measured twice. The sample with children that had no prior computer experience showed that in the first week there was no real significant difference in between the means of GUI (98.5 sec.) and TUI (105 sec.). The second week showed very similar results. Through formal observation, they claim that children seem to be engaging both interfaces.

The ‘wow’ factor or the novelty effect seems to be diminished when GUIs and TUIs when they are exposed for the first time.

This type of approach seems to be useful when trying to understand user’s engagement and extend it as a comparison method in between TUIs and GUIs. By using observation methods, for example recording facial expressions when recording tasks, can provide extra information to the analysis performed. Furthermore, the authors warn that facial expressions might be prone to subjectivity.

User perspectives on mixed reality tabletop visualization for face-to-face collaborative design review

Xiangyu, Wang, and P. S. Dunston. 2008. “User perspectives on mixed reality tabletop visualization for face-to-face collaborative design review.” Automation in Construction no. 17 (4). doi: 10.1016/j.autcon.2007.07.002.

There is a difference when working with Virtual Reality environments, a technology originally embraced by architects for project presentation, or Mixed Reality managed to combine the VR with real world elements. As a variant of MR, AR provides a different insight by enhancing computer generated information to objects, thus changing the real world experience.

This is a project where the researchers investigated collaboration differences from users when working with AR/MR and paper-based plans for architectural models. Their main objective is to understand the limitations that these work environments will encounter when teams work on a problem-solving task.

Figure 6. Paper based vs. AR environment

Spatial cognition, another method of assessing, is the ability to know or learn the physical layout of things, a way in which individuals comprehend and function in reference to space relations. It is important to identify psychological issues in human’s comprehensions of architectural drawings and models.

They used sixteen graduate students, 12 men and 4 women. They were distributed in 8 groups of 2 each.

They provided two separate designs for the MR and the paper-based environment. The Subjects were asked to find errors on the design, and were provided with enough time to find and memorize such design errors. The design plans were very similar layouts. The subjects were expected to find on each design. They could be found when combining both design layout or by analysing them independently.

 Figure 7. Example of design error.

Each session lasted from two to three hours, including a training session and a post-experiment questionnaire session.


They utilised questionnaires to analyse what influence MR has on the performance and perception of the subjects. There were other more extensive results not mentioned in this paper.

There were three questionnaires designed to investigate user’s experience.

Experience questionnaire for comparing MR and paper based method.
Developed on a six point scale representing poor and excellent. Included questions: Quality of visual representation, physical comfort, level of immersion, ease of navigation and ability to maintain sense of location and orientation. A one-way analysis of variance (ANOVA) was implemented on the individual ratings for both cases.

Results: The increased 3D perception, spatial layout and cognition of the MR system allowed the users to offload some of the mental processing, allowing them to fulfil the tasks in a shorter time. On the other hand the paper based system presented to be more comfortable than the MR. The MR system presented some issues with the participants’ mobility due to cables and bulky head-mounting displays.

• Attitude questionnaire. Used to collect subjects’ opinions regarding the value of MR system for collaborative work. One questionnaire for each interface.
They included issues like communication, problem-solving, communication, contribution and comprehension. The questionnaires were in a 4-point rating scale answers using “totally agree”, “agree”, “disagree” and “totally disagree”. The research team them decided to average questionnaire 1 answers of ‘totally agree’ with the corresponding statement of questionnaire 2. They argue this made the interpretation and analysis process easier.

Results: 57% of the sample felt that it was easier to comprehend the 3D environment when using the MR system. 75% believe they had the MR system facilitated problem-solving. One interesting response was that 62% agreed that the MR system better facilitated collaboration tasks over the paper system when on the other hand 69% felt that the communication process was better when using the paper-based system. In general, it was evidenced that the MR assists individual performances but not necessarily supports collaborative work. They argue that usability could be a critical factor for this.

• Usability questionnaire. It was used to investigate questions not properly addressed on the previous questionnaires. Some issues covered disorientation, performing difficulties, control over the system, nausea, comfort, response and long-term use. This questionnaire used a six-step scale where 1 stand for very little and 6 for very much. There were 19 usability questions covering the issues just mentioned. The questions meant to answer questions of how the MR system influenced the user’s performance, meanwhile other ones focused on users’ ‘comfort’

Results: The users felt a little disoriented and reported that the head mounting display created difficulties for performing. It had to do with the bulkiness and size of the HMD, they even had to use one hand to hold it. The users did not had any eye fatigue, in the experiment usually the subjects used the HMD for around 20 – 30 minutes for the entire experiment, nevertheless, they did experienced little nausea during the experiment.   81% of responders think they will not be reluctant to use similar MR systems in the future.

Quantifying the performance of subjects is a difficult task, because performance is not well defined. There are issues like the naturalism of the interaction and the degree of presence that the user feels. Usability-related issues such as ease of use and user comfort will also be important.

The impact of tangible user interfaces on spatial cognition during collaborative design

Kim, Mi Jeong, and Mary Lou Maher. 2008. “The impact of tangible user interfaces on spatial cognition during collaborative design.” Design Studies no. 29 (3):222-253. doi: 10.1016/j.destud.2007.12.006.

This is a project that focused on understanding the impact that TUIs have on spatial cognition. As a interior design task, this experiment utilises 3D models of furniture through AR on a table-top system against a traditional 3D CAD software. The TUI was built using ARToolkit. The table-top system was constructed with a horizontal display acting as a table plus a vertical display instead of using a HMD.

The task for the experiment presented a small-scale space planning problem, in this case a home office.  The main focus was to understand how student perceived space, objects and their relation with each other. In this case they experimented with very similar design spaces for TUI and GUI.

Method (Design)

The subjects involved 2nd year students experienced with CAD systems. There were 10 students grouped in pairs. Each pair experimented with both systems in a 20 minutes for each system. Each experiment was done in a day. Subjects were not asked to hand in a final design since the researchers were investigating their behaviour.

They used protocol analysis to analyse the spatial cognition of collaborative design. For this, they were given a small-scale space-planning problem using furniture.

Method (Analysis)

The researchers utilised protocol analysis to analyse the subjects’ design behaviour. Acording to the author, protocol analysis is the best method to analyse problem-solving processes in design studies. There are four steps in the methodology: protocol collection, segmentation, coding and analysis.

• Protocol collection

Akin (1986) mentions that a protocol is the activity performed by the subject to solve a problem. This is commonly presented through sketches, notes, video or audio recordings. More modern protocol studies focus on how the activities take place. It analyses events such as actions or like in this case arranging elements, to produce a clearer understanding of the subjects’ design process.

There are two types of protocols: Concurrent protocol focuses on the process oriented aspect of design. In this case ‘thinking aloud’ is a technique that can be used. Retrospective protocol is focused on user-oriented processes. This is commonly applied by asking the subject to reflect and share his task experiences.

• Coding Scheme

The author followed and then adapted a coding scheme developed by Suwa (1999). This coding scheme is divided into segments according to “subjects’ intentions”. Originally these categories are:

  1. Physical: Actions that have a direct representation on paper. They can be drawing-action and looking-actions.
  2. Perceptual: They relate to visual/spatial elements of the produced images.
  3. Functional: Relation of meaning, functions or abstract concepts linked to the previous two categories.
  4. Conceptual: They are elements that involve higher cognitive elements such as aesthetic evaluations, goals or retrieval of knowledge.

For this the research team generated their coding scheme where they adapted new items to the original list.

Level Description
Action Level
3D modelling actions
PlaceNew Place a new object on the plan from the library
Rotate Change the location of an existing object again
Perception Level
Perceptual Activities
E-visual feature Attention to a visual feature (geometric or physical properties) of an object or a space
E-relation Attention to a spatial relationship among objects or spaces including orientation
Process Level
Set-up Goal Activities
G-knowledge Goals to introduce new functions derived from explicit knowledge or experience
G-implicit Goals to introduce new functions in a way that is implicit
P-space The features and constraints required for a design solution
S-space The features and behaviours of a design solution
Collaborative Level
Cognitive synchronization
Proposal Propose an opinion of the problem or a new idea in the solution space
Argument Support or oppose the opinion or the proposed idea
Gesture Actions
Design gesture Large hand movements above the 3D plan
Touch gesture Touch a 3D block with hands or the mouse

This table provided some examples of codes that can be utilised to analyse the different categories. The researchers still created a combination of codes in order to identify different behavioural procedures in the different systems.

Results (Observations)

• Behaviour

Based on scripts and video analysis, two coders collectively retrieved all the verbal narrative. The reliability of the coding process was measured by calculating the Kappa values of both coders.

There were different approaches to the design brief according to the system they were using. TUI users focused on more abstract levels of the design, meanwhile GUI users approached directly to the library of models based on the information provided by the brief. TUI users discussed ideas by moving objects where TUI users discussed it verbally. Both TUI and GUI users pointed to an object by touching it. The difference is that when touching an object with a mouse does not do anything.

GUI sessions were longer than the TUI sessions. The author mentions that TUI session had much more progress even when both sessions were assigned the same amount of time.

• Spatial cognition

This analysis was analysed by using the encoded protocols and produced the occurrence percentages of cognitive-activity categories to examine differences between the two sessions. The statistical analysis of they performed a Mann-Whitney U test.

TUI users managed to produce more interpretations of the space. TUI sessions produced more sudden ‘insight’. When using GUIs, users generally completed the action once the mouse was released. TUI users were capable of identifying deeper ‘furniture’ functionality. TUI users formulated the problem in different ways and came up with different ideas for solving the problem.

• Collaboration

TUI users produced more arguments in the design process. TUI users presented more ‘external representations’ using hands and arms to communicate with team members. The author argues that these gestures may assist in complementary strategies of 3D modelling actions.


Tangible User Interface for Chemistry Education: Comparative Evaluation and Re-Design

Fjeld, Morten, Jonas Fredriksson, Martin Ejdestig, Florin Duca, Kristina Boetschi, Benedikt Voegtli, Patrick Juchli, and Acm. 2007. “Tangible User Interface for Chemistry Education: Comparative Evaluation and Re-Design.” Conference on Human Factors in Computing Systems, Vols 1 and 2:805-808.

This is a project that involves AR and chemistry education. The interface is called Augmented Chemistry. It is an attempt to provide a TUI version of the traditional ball and stick system traditionally used in secondary school chemistry education. Users use an AR system to create compound elements based on the periodic table by combining molecules which are virtually picked up from a book, by rotating the AR cube marker, the user can decide where is the molecule will be implemented.


There was a process of Comparative Evaluation to determine whether AC can be successfully implemented in chemistry lessons in secondary school level. They compared learning effectiveness and user acceptance.

Twenty six biology and laboratory students from secondary school. There were 5 women and 21 men. The experiments were organised on a AB-BA design. They were divided into two groups of equal size and worked individually with both systems for two weeks. Group one worked with AC for the first week meanwhile group two worked with the stick and ball system; the week after, they exchanged systems.

The exercises were arranged this way:

They studied users’ mood, mental task load, physical task load, satisfaction, perceived usability, and system preferences by using questionnaires and then converted into an assessment of user acceptance.


The BS solved more problems than the AC system, but still both systems promoted same level of information retention. The evidence suggest that the AC system does not provide a more efficient environment for learning. Many of the element configurations are easier to manage on a fully physical object, where size, element labelling and system parameters are more suited.

Learning with TUIs

A cube to learn: a tangible user interface for the design of a learning appliance.

Terrenghi, Lucia, Matthias Kranz, Paul Holleis, and Albrecht Schmidt. 2006. “A cube to learn: a tangible user interface for the design of a learning appliance.” Personal and Ubiquitous Computing no. 10 (2-3):153-158. doi: 10.1007/s00779-005-0025-8.

This paper describes work developed for a cube with six displays where information is displayed. It is combination of hardware and software. By the use of multiple-choice test system this application can be used for several tests or games. In this case it was used for a language translation test where on the top face the original word appears and on the remaining 5 faces there are multiple choices where one is the correct answer. When facing the correct side (answer) and shaking the cube, it provides the result.

The team argues that “breaking conventions of how a computer looks like and providing children a playful interface” it will be easier to introduce these type of interfaces into an everyday activity. Children do not see an apparatus like this like a traditional learning tool but rather as a toy.


Applying tangible story avatars to enhance children’s collaborative storytelling.

Liu, Chen-Chung, Kuo-Ping Liu, Pi-Hui Wang, Gwo-Dong Chen, and Mu-Chun Su. 2012. “Applying tangible story avatars to enhance children’s collaborative storytelling.” British Journal of Educational Technology no. 43 (1):39-51. doi: 10.1111/j.1467-8535.2010.01146.x.

This project describes the use of avatars to facilitate collaborative storytelling with children. In this case the avatars are physical marionettes.

They describe related works that include:

• Virtual storytelling environments.

StoryBuilder and FaTe2 where the use of the Web allows to share their stories with other users.

• Software avatars.

It is necessary to equip avatars with an intuitive manipulation interface that can facilitate behaviours and enhance social interaction to improve interpersonal communication skills.

• Tangible Storytelling Avatars (TSA)

TellTale, Storymat and Dolltalk use voice recording functions to assist children in the creation and sharing of their stories.

With TUIs it is important to analyse how is the social interaction going to take place to be able to provide an environment where they can help each other.

Storytelling systems: constructing the innerface of the interface

Umaschi, M., and J. Cassell. 1997. “Storytelling systems: constructing the innerface of the interface.” In Proceedings Second International Conference on Cognitive Technology Humanizing the Information Age, edited by J. P. Marsh, C. L. Nehaniv and B. Gorayska, 98-108108. IEEE Comput. Soc.

The researchers identify the necessity of computational tools that are specifically designed to encourage exploration and communication issues.

By building things, learners are capable of producing more and more different type of ideas.

This project SAGE, uses personal connecitons, where children program their interactive storyteller according to their own culture and interests. SAGE encourages constructivist learning.

Their embodied objects are cover stuffed animals with computational affordances. Psychologist Winnicot refers to this as a transitional object where it becomes the first object “not me” for children.


The CTI framework: informing the design of tangible systems for children

Antle, Alissa. 2007. The CTI framework: informing the design of tangible systems for children. Paper read at Proceedings of the 1st international conference on Tangible and embedded interaction, at Baton Rouge, Louisiana.

There are five themes analysed in this paper.

  1. Space for Action.
    1. Tangible interfaces must use space.
      • Design Concepts: Body-based interaction and control. Age appropriate repertoire of physical actions. Existing performative knowledge.
    2. Pragmatic actions involve manipulating physical entities to directly accomplish a task. Lego and Tetris provide understanding of how connecting systems work.
  • • Design Concepts: Epistemic actions and offloading difficult mental processes.
  1. Perceptual Mappings
    1. Perceptual Affordances

Ways that the children will activate the interface or actions required to implement

  • • Design Concepts: Perceptual affordances and Designed affordances.
  1. Scales of Experience and Representation

Relating real world objects and their scales in their interpretation of objects may be an issue.

  • • Design Concepts: Child-body scale as reference
  1. Behavioural Mapping
    1. Cognitive Mode Switching

The process of experimenting with concepts and structures through the interface relates to children’s metacognitive strategies. Their ability to self-regulate and reflect.

  • • Design Concepts: Experiental vs. reflective modes of cognition. Ready-to-hand vs. present-at-hand.
  1. Cause and Effect

Children as young as three can rely on temporal order when identifying causes. Several variables may influence simple casual judgement including: the degree of similarity between causes and effects and the inhibitory nature of the causal relations.

  • • Design Concepts: Temporal precedence. Covariation. Temporal and spatial contiguity.
  1. Semantic Mappings
    1. The Problem with Multiple Representations

Children under sever may have difficulty relating physical manipulatives to other forms of representation across context.

  • • Design Concepts: Reveal representational mappings. Exploration of relationships between entities and representations.
  1. Mapping between Representations

It is necessary to relate abstract ideas to concrete ideas. As children grow older, their spatial cognition changes. This improves their understanding of their own space and position.

  • • Design Concepts: Reciprocal mappings between physical and mental representation
  1. Grounding Understanding in Body-based Schema

Their process of solving kinetic problems abstractly and concretely.

  • • Design Concepts: Leverage children’s understandings of bodily-based concepts to help them understand abstract concepts.
  1. Grounding Understanding in Spatial Schema

Children use rich spatial schemata as foundation for the development of other concrete and abstract schemata.

  • • Design Concepts: Leverage children’s understanding of concrete spatial schemata to help them understand abstract concepts.
  1. Space for Friends
    1. Collaboration

Most GUIs are constrained to a single user interaction. TUIs can allow collaboration from several users.

  • • Design Concepts: Interaction supports rather than requires collaboration. Multiple input units or modes. Protocol for changing mode or transfer of control.
  1. Imitation and Intentional Affordances

Imitating actions from a guide or an authoritative figure can allow children to understand how to use tools.

  • • Design Concepts: Clues to intentional affordances. Visual access to performative actions. Turn-taking of physical or spatial controls.

Magic Story Cube: an interactive tangible interface for storytelling

Zhou, Zhiying, Adrian David Cheok, JiunHorng Pan, and Yu Li. 2004. Magic Story Cube: an interactive tangible interface for storytelling. ACE 2004, 2004, at Singapore, Singapore.

This is a project where a multi-faced cube is utilised to narrate stories from the Bible. By combining pre-defined sequences on to each face of the cube, the storytelling process takes place. This may be handy when required to tell a story before some activity still keeping the game factor.

They argue that presenting the narrative this way appears to be more attractive and understandable to children.

Children’s Storytelling and Programming with Robotic Characters

Ryokai, K., M. J. Lee, J. M. Breitbart, and Acm. 2009. “Children’s Storytelling and Programming with Robotic Characters.” C & C 09: Proceedings of the 2009 Acm Sigchi Conference on Creativity and Cognition:19-28.

This project combines GUI with TUIs. By utilising a robot, in this case a dinosaur, the children can relate more to the activities provided. They manage to translate into the mind of the dinosaur which then starts some actions. Enriched drawings are programmed to control this robotic character.

This way events can be shared with others and they explore possible outcomes of different events and negotiate them with other users. Children manage to create abstract concepts and create meaning between these ideas.

By combining several sensory elements in the interface, there is no exclusion on a medium over another. This way every children can use different engagements to represent their ideas.

There are some robotic elements that allow the incursion to these type of interfaces, including: Lego Mindstorms, WowWee and Ugobe’s Pleo.  The problem with using interfaces like this is that it requires high level of technical competency.

In storytelling terms, there are some tools developed KidPad for example allows generation of storytelling based on children’s drawings. StoryMat records oral stories and movements of stuffed animals on an augmented play.

This project uses a transparent table, underneath a camera is placed to detect the markers. This may be handy when using tangible objects that require a marker.


Storytelling and TUIs

Embodied metaphors in tangible interaction design.

Bakker, S., A. N. Antle, et al. (2012). “Embodied metaphors in tangible interaction design.” Personal and Ubiquitous Computing 16(4): 433-449.

This is a paper that analyses the use of metaphors for embodied interaction or tangible user interfaces (TUI).

It is required to study which embodied metaphors will be able to be translated into the TUI. When children employ embodied interaction, it supports their cognitive development. Their knowledge is developed by experiences they had in the real world. High-order thinking processes arises from bodily disposition and experiences producing this way their cognitive structures. These thinking processes are also known as ‘image or bodied schemata’. Here is where metaphors make relationships in between concepts.

Researchers like Vygotsky and Galperin, have stated how material operations are the source of mental procedures. For this, physical objects can narrow the gap in between concrete and abstract concepts as proposed by Bruner. So the process of gaining new knowledge is based on experiences that are further on reflected. TUIs specialise in this kind of reflection, specially abstract problems.

On the design aspect, it is very important that the interaction design is properly developed, making sure that the embodied object translates into the cognitive process.

This team identified five stages where the object can be analysed to provide the best user involvement:

  1. Enactment studies to identify applicable embodied metaphors
  2. Creating low-fidelity prototypes based on embodied metaphors, to explore the input design space
  3. Evaluating low-fidelity prototypes to validate the input design space in terms of affordances which support embodied schematic movements
  4. Creating high-fidelity interactive prototypes with suitable affordances, to explore the mapping between the input design space and metaphorically linked output responses
  5. Evaluating high-fidelity interactive prototypes, to validate the input design space, embodied interactional mappings and output responses

Applying Situation Awareness Approach to Cooperative Play in Interactive Installation Storytelling System

Wen-Hwa, C., S. Chich-Jen, et al. (2010). “Applying Situation Awareness Approach to Cooperative Play in Interactive Installation Storytelling System.” Cooperative Design, Visualization, and Engineering. Proceedings 7th International Conference, CDVE 2010.

Jean Piaget had previously identified the important role in cognitive development when playing. When learners utilise their cognitive thinking process to generate new knowledge, they construct, reconstruct and organize their knowledge. This has been identified as a process for constructivist learning. Furthermore, social constructivism involves a system to share information and uncertain assignments. Children can be able to share their understanding of specific tasks and through the process of peer learning they can become active participants.

It is suggested that it is necessary to produce correlation in between the TUI and the GUI to avoid cognitive differences when designing the interface. Providing a simple interactive process will enhance the awareness of the user.

It is necessary to make a visual, and tangible relationship in between the tools applied in order to make them more intuitive.

Using avatars and virtual environments in learning: What do they have to offer?

Falloon, G. (2010). “Using avatars and virtual environments in learning: What do they have to offer?” British Journal of Educational Technology 41(1): 108-122.

Peterson (2005: p.1) has pinpointed how interaction in virtual spaces can be improved by the use of a ‘virtual self’. These virtual selves are ‘online manifestations of self’. This visible persona acts as a vessel that allows imaginary experiences to transcend to the actual experiences and vice versa.

There are many people who have trouble engaging in face-to-face communication, virtual scenes can facilitate these social interactions. Virtual environments are pathways where people can communicate in way in which they would not be capable to do so in a different setting. Not only they can facilitate the communication process but also they can evolve to the requirements of their users. This potential for adaptation will also enhance social interaction. The potential presented by virtual environments can provide grounding for collective learning in a way that would not be achievable in a traditional setting.

The debate of how learners will engage with virtual environments, it has been identified how they provide a ‘safe pedagogical platform’ for learning. Meanwhile in learner-centered tasks, avatar environments have been identified useful.

The digital world can promote student communication. Seymour Papert has identified how creating Microworlds has allowed learners to organise their thoughts in a way that they can be conducted efficiently. This concept shares many of the elements of a virtual world.

Evidence suggests that by emulating real world environments where the learner can pinpoint specific questions or problems will enhance the learning process. Also, by communicating their findings will promote their proficiency to collaborate with other peers.

Micro Culture: Interactive Storytelling and Learning in the Museum

Marchetti, E. (2012). “Micro Culture: Interactive Storytelling and Learning in the Museum.” 2012 IEEE 4th International Conference on Digital Game and Intelligent Toy Enhanced Learning (DIGITEL 2012).

There is a necessity for learners to express themselves within a learning platform.

Several projects have been developed where playful interaction has been utilised in order to provide better engagement with knowledge.

Hall and Bannon developed through ‘user-cantered iterative design process’ where researchers could analyse the learning needs of children in the Hunt Museum in Limerick, Ireland. In the Berlin Museum of Natural History, a table-top installation called The ‘Tree of Life’ was developed to provide visitors with other means of information. The interface of Kurio, presented a role-play activity where families interacted with tangible objects and a table touch interface. Role-play dynamics can promote storytelling proceedings. Visitors presented more interest about lives of people from the past. MicroCulture is another project where children engage through role-play with roles of political authorities when taking control of geographic territories.

When employing knowledge from adult experts when they are provided with problem-solving activities, children can pick up information by combining all their resources in a non-formal domain.

Interactive Storytelling for Elementary School Nature Science Education

Jui-Feng, W., K. Hsiu-lien, et al. (2011). “Interactive Storytelling for Elementary School Nature Science Education.” 2011 11th IEEE International Conference on Advanced Learning Technologies (ICALT 2011).

In education, it has been proven that storytelling is a good way to promote thinking within the students. The thinking framework can be displayed thus providing a positive approach to conceive abstract perceptions are acquired through storytelling. Learning through storytelling usually requires a person that assesses the students. This could be problematic and time consuming. To solve this, the storytelling process can be divided into stages where the supervisor can provide guidance or feedback.

Interactive storytelling for children education

Evangelista, C., V. Neri, et al. (2009). “Interactive storytelling for children education.” 2009 Conference in Games and Virtual Worlds for Serious Applications (VS-GAMES). First International Conference, VS-GAMES 2009.

When utilising storytelling in education, the experience or event can told digitally through electronic means. According to this research, seven main components are identified for the structure of a story:

“Point of view (stories should be personal and “authentic”), The Dramatic Question (stories should tell something worthy), Emotional Content (stories should involve

people), The Gift Of Your Voice (a way to personalize stories), Soundtrack (which helps understanding and adds aesthetical value), Economy (very often stories can be told with few words, images and music: let the “implicit” talk, use metaphors), Pacing (good stories “breath”).”

Through the constructivist approach, every learner will be capable to build from the external world, his or her own cognitive parameters. For these type of scenario, games can act as an apparatus for children in order for them to prompt their curiosity and motivation.

Another way for learners to share their stories is through drawings. Children reach an intellectual stage where they learn to control their gestures becoming capable of expressing feelings and emotions through illustrations. They can express personal problems and anxieties in a clearer way. Eventually it becomes a mindful communication operation that manifests in kids in between 10 and 13. Technology can help in the elaboration of these visual stories.

G-Flash: An authoring tool for guided digital storytelling

Jumail, D. R. A. Rambli, et al. (2011). “G-Flash: An authoring tool for guided digital storytelling.” 2011 IEEE Symposium on Computers & Informatics (ISCI).

Many educators are becoming more eager to implement a more interactive method in their teaching activities, including constructive learning and collaboration through edutainment. Digital storytelling is a powerful tool that allows elaborated stories to be created with a logic sense. Combining many types of media can produce digital storytelling.

Illustrated flash cards for example, can improve memory and recall. They can be utilised as an assistant to develop the story. This might prove useful when lacking the presence of a guide in the storytelling process.

Several storytelling tools have been developed. For example: KIdPad, a collaborative storytelling tool, Wayang authoring from Bremen University developed an authoring tool with graphical characters, oTTomer, developed to present a story to young children in between 6 and 12 and the FaTe2 where they combine storytelling with edutainment on a web-based, multiuser virtual space.

Designing for Children’s Mobile Storytelling

Franckel, S., E. Bonsignore, et al. (2010). “Designing for Children’s Mobile Storytelling.” International Journal of Mobile Human Computer Interaction 2(2).

Since storytelling has proven useful for education purposes, mobile tools have been studied to understand how do children want to tell their stories once engaged in a mobile platform.

Children have integrated mobile platforms to their everyday life, even from a very early age.

When it comes to human communication and making sense of concepts, storytelling is a vital. In the case of children, it provides them with tools for storytelling. It provides a fundamental pathway to explore cultures.

According to this research although much research has been done around storytelling, not much of it has been done in how mobile platforms can develop storytelling means. There has been many cases in which the digital platform behaves just as an extension of the print based information. Other tools communicate stories but do not allow those stories to be extended or to create new ones.

Technology and perspectives for AR and TUIs

ARBlocks: A projective augmented reality platform for educational activities

Roberto, R. A. and V. Teichrieb (2012). “ARBlocks: A projective augmented reality platform for educational activities.” 2012 IEEE Virtual Reality (VR).

ARBlocks is a project that uses the playfulness activities as the main approach to teach a topic. Through playfulness activities, the user engages without noticing he or she is learning. It is identified that engagement, accessibility and group learning is promoted when using TUIs.

ARBlock uses uses Voxar Labs for optical tracking. Voxar Labs allows the tracking to be marker-based or marker-less.

One of the best ways to enhance HCI and intuitiveness in interfaces is through natural interaction.

Voxar Labs focuses on research of natural interaction, augmented reality, visualisation and tracking.

TUIs vs. GUIs: comparing the learning potential with preschoolers.

Sylla, C., P. Branco, et al. (2012). “TUIs vs. GUIs: comparing the learning potential with preschoolers.” Personal and Ubiquitous Computing 16(4): 421-432.

TUIs have big potential to promote learning. They are capable of providing long-lasting involvement making children to have a better engagement. Papert has noted through the constructionist approach the importance that physical objects have on the construction of knowledge. Also, Maria Montessori developed a method where children learn by playing with tractable objects. Further on Froebel-inspired Manipulatives combined the Montessori method of learning abstract concepts with the possibility of modelling of objects of the real world. For this reason groups including Lifelong Kindergarten at MIT Media Lab or the MIT Tangible Media Group have been involved in the use of TUIs to assist learning.

The period of graphic communication starts around the age of 4, when they enter the Preschematic Stage. They try to produce objects that relate to their environment. Realistic likeness is not relevant at this stage it would actually interfere with their conception of the object.

Some elements show relevant to children. The use of animation and tangible tools and the novelty effect improve the engagement that children have. Furthermore TUIs provide a long-lasting engagement that promotes learning. Meanwhile GUIs lack the richness of human senses utilised in the real world limiting this way their communication channels.

The use of single image to communicate what they were capable to remember was a good approach since they describe all the elements that they managed to retain or the elements that most impressed them. This makes it easier to understand the impact the interface had on them.

 Tangible user interface of digital products in multi-displays.

Lee, J. Y., M. S. Kim, et al. (2012). “Tangible user interface of digital products in multi-displays.” International Journal of Advanced Manufacturing Technology 59(9-12): 1245-1259.

The feedback provided by AR technologies creates the effect of natural interaction. Ubiquitous and mobile environments through mixed reality can enhance learning.

Several inexpensive optical tracking devices like the WiiMote can provide a good solution for implementation.


Editorial: the evolving field of tangible interaction for children: the challenge of empirical validation

Zaman, B., V. Vanden Abeele, et al. (2012). “Editorial: the evolving field of tangible interaction for children: the challenge of empirical validation.” Personal and Ubiquitous Computing 16(4): 367-378.

Improved usability has been related to the implementation of tangibility. It allows utilizing the intuitive skills of the users. TUIs are considered to be more enjoyable than traditional interfaces and allow collaboration. Some superior attributes of TUIs are its explicit input devices, and improved accessibility, also spatial use with the control of a physical object. GUIs usually require more steps in interaction which makes them slower and less efficient compared to TUIs.

With the use of epistemic activities the search for problem-solving is achieved. On the other hand, pragmatic ones will bring the user physically closer to the objective.

Previous experiments have shown that TUIs provide physio-pleasure. The importance of sensory reactions is related to user’s immersion.

When working in collaborative environments, tangible object have different qualities that may put them out of reach or blocking their access making collaborative negotiation different.

When analysing the interface, it is necessary to perform a comparative study. Thorugh this process, there is a comparison of the different interactivity styles on a single object. Evaluation studies will compare the interactivity in between different versions.