Querying Europeana Linked Data through Tangible User Interfaces

Querying Europeana Linked Data through Tangible User Interfaces

Basic Queries

After building essential queries to be performed, I have now designed the final objects to be implemented on the interface. For ease of build, I decided to use basic cubes. I believe users will find easier to build cubes than more complex shapes.

Data providers and object types
Data providers and object types

All these query elements are tied to Europeana’s Data Model thus allowing users to indicate the particular set of metadata that they want to find. Compared to my previous report, I have now removed the¬†type of object¬†from the queries. The User Centred Design study showed that not many people were particularly focused on the particular¬†type¬†until they got some results, thus making it a secondary tool. Nevertheless, I have still provided a display window of the amount of results retrieved per¬†object type¬†and¬†provider.

When the query is entered, the list autoscrolls showing the user the whole list of providers. When a new query is entered, the list is updated and presents the new list to the user.

Users can choose the amount of results that they want to select. I have provided a pre-designed set of choices. These are: 12, 24, 48 and 96 results. Users choose these variables by rotating the pyfo. If the pyfo is removed, the last chosen option stays active.

The¬†what, where¬†and¬†who¬†pyfos will still have the option to add the OR or NOT boolean option. This is performed by rotation the main pyfo +or- 45¬ļ as indicated in the design.

Rotated pyfo
Rotated pyfo

Testing The Table

I have also finished building a small version of the prototype table. At this stage I am just finalising final technical details to make the markers more stable. They are still flickering to some degree. This might affect the interaction when testing the interface.

I designed a search and delete fiducials on a single pyfo (cube) object that will be used to enter or delete the elements on the screen.

Here is a video of the current state of the interface:

Tangible Interaction with Sifteo Cubes.

 Dynamic Fiducial Pyfos (DFPs)

During the last weeks I have started developing what I would like to call DFPs short for dynamic-fiducial-pyfos. With the help of some friends I have now a basic skeleton to extend my interactive experiments. Here is a video of the result:


Tangible Interaction and Pyfos

After I submitted my upgrade draft, I realised that I was going to encounter some issues when working with pyfos when using them as part of the fiducials for the TUIO system. Since users have to combine different concepts (e.g. Roman + pottery or painter + 1800) this will result in a numerous amount of pyfos. The interface already has several objects that can not be removed since they are part of the basic interactions such as: map navigation, box dragging, etc… Therefore, I decided to explore a little bit further. I need to find a way to extend the capabilities of pyfos.

Pyfos have three main states: token, constraint and token+constraint. The TAC (Token and Constraints) paradigm in Tangible User Interfaces (TUIs) offers a set of constructs of how these objects react. Nevertheless, it can be argued that technology can offer pyfos that can self adapt or expand the constraints that bound them. This due to the fact that physical objects such as pyfos cannot morph or change according to user needs.


Other researchers are exploring how these TAC approaches can be expanded. It is this search that directed me to explore with mini-displays and sensors. Since I had worked with some Internet of Things and Arduino, I thought of designing some display that detected two different RFIDs to make the combinations and display a specific result. That result display presents the final combination in a form of fiducial so the TUIO interface can detect it. This way, users can pre-design a concept combination thus integrating it in a final dynamic-fiducial-pyfo (DFP) that encompasses that prior combination. Most importantly, DFPs can also produce other display results on the table-top without clustering tools or options.


Producing DFPs

There are some alternatives to produce¬†DFPs out there. Many of them require to be built from scratch, but there are some products that can be adapted to our needs without having to fiddle or hack any of the electronics. Although there is a wide variety of tools out there, these are the most ‘approachable’ tools that I encountered.


Since I had worked with Arduino and Internet of Things before, I thought of building a mini-display with either RFID or any other type of sensor. I found the Educubes project that presented a good opportunity to start developing for this idea. This might also prove to be beneficial since there are some TFT mini-displays that support touchscreen actions as well.

Educubes by Adafruit
Educubes by Adafruit

 Although this presented a good opportunity to develop, I needed to start producing tools that could work straight away instead of focusing on the electronics. Moreover I thought that the size of the electronics is still quite big for them to be used on the prototype. But it was mainly the issue of working with a wide range of electronics and hacking them so they can do what I wanted.

Sifteo Cubes

After searching I encountered Sifteo Cubes. These cubes already provide a very nice presentation that encompasses a wide range of electronics and a mini-display. Moreover these cubes can be programmed through an SDK provided by the same company. I decided to jump ahead and ordered a second generation Sifteo Cubes.

My surprise was at the moment of using the SDK. I was not the first user to be put down by its complexity. The Sifteo SDK works with C++ with other command tools to run installations and device management. Moreover, through the forums I encountered that the released SDK contained some bugs, which made some of the tutorials not to work.

Nevertheless, I encountered some compiled SDK in GitHub such as Investio and Sifteo Blickets. They provided me with some hints into how to actually start using and managing the cubes. I still had to learn how to program what I needed. So I started first learning how to make interactions. There is a base of few interactions that are supported by the sensors in the cube that include: tilt, pair, shake and press. 

Sifteo Actions
Sifteo Actions

Although it seems quite nice in pictures, the task was not so simple. Since I do not come from a programming background, working with C++ was a huge challenge. First I did a tutorial on neighbouring. Here is the video:

A few days after, I started working with other actions such as tilt and press. It is relevant to mention that I worked using the examples that came with the SDK so, the interactions were pretty much pre-designed and I was just learning a few basic commands that might be used. Here is a video of this stage.

The problem started when working with my specific requirements. I needed an array of options per cube that could be combined between them. This so the final combinations could be applied in the Europeana TUIO system. Using C++ this was not so straightforward. In a nutshell, this is what I needed to create:

Basic combination skeleton
Basic combination skeleton

It took me a lot of time and effort to find a way to program this interaction. I could program something like this with other languages but not with C++. Therefore I asked for some help to develop this. Kevin Lesur from One Life Remains gave me a hand with this. So this is was the basic skeleton built for the interactions:

Operations and interactions skeleton.
Operations and interactions skeleton.

Two cubes are required to make the combination through neighbouring. When they are combined a third cube presents its combination that will eventually show a fiducial making it a DFP. To navigate between the cubes options, users can tilt the cubes in either direction.

This way I am hoping to now carry on and go back to the TUIO experiments and see how these DFPs work on the tabletop system.


Draft Almost Ready

Working Toward my Final Draft

Three weeks ago I had a meeting where the original index was approved. I kept on working on it and I have what it the closest thing to a final draft version of it. This was built from previous reports and presentations in conferences and workshops that I have made. There is still a lot of work to do in my research but this encompass the theoretical foundations for my hypotheses.

The Content In My Thesis

There is mainly six chapters. Starting from the Introduction where I discuss the origin of the problem and the different theoretical (and practical) approaches to provide a solution. The second chapter discusses how different cultural heritage (CH) organisations work on the WEb and built what I call the Online Cultural Heritage (OCH). РPaper coming soon! РThe third chapter discusses how interfaces work and its types followed by different testing methods such as Usability, UX and Engagement in Chapter 4.

The interface prototype is presented in Chapter 5. Then I discuss what is the future work that I need to do to build, test and implement the interface in Chapter 6. I also have a conclusion and references chapter but to get an idea of my research based on the index, it can be done solely on the first six.



1. Research Introduction 
1.1.    Introduction
1.2.    Cultural Heritage Institutions
1.2.1.    Museums.
1.2.2.    Libraries and Archives
1.3.    Museums, Libraries and Archives: Online Cultural Heritage (OCH)

2. Online Cultural Heritage 
2.1.    Reshaping the OCH Technology to Engage with Information
2.2.    Producing Common Ground. Information in OCH
2.2.1.    The Data Information Knowledge Wisdom (DIKW) Model: Knowledge Construction
2.2.2.    Information and its users   Information Production   Information Sharing   Engagement with Information
2.3.    Users First Approach to OCH                                                                                                               2.3.1.    Information Engagement
2.4.    Information Production, Sharing and Engagement. Information stages in OCH context
2.5.    Current State of Engagement with Information
2.5.1.    Content Management Systems
2.5.2.    Data Querying Systems
2.5.3.    Technology

3. User Interfaces 
3.1.    Introduction to User Interfaces
3.2.    Tangible User Interfaces
3.2.1.    Design Principles (HCI) in TUI Context
‚ÄĘ Visibility
‚ÄĘ Feedback
‚ÄĘ Mapping
‚ÄĘ Affordances
‚ÄĘ Pyfos ‚Äď Token and Constrains Paradigm
‚ÄĘ Physical Objects (Embodiment)
3.3.    TUIs in OCH Context
‚ÄĘ TUIS and Data Exploration, a Constructivist Approach

4. Interaction Design Methodologies and Measurements
‚ÄĘ SA&D and HCI Development Methodology
‚ÄĘ Software Development Methodology
‚ÄĘ Design and Graphic Design Methodology
4.1.    Integrating Methodologies
4.2.    Testing and Measurements
4.2.1.    Usability
‚ÄĘ Testing
‚ÄĘ Evaluation ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† ¬† 4.2.2.¬†¬†¬† Engagement
4.2.3.    Design
4.3.    Evaluation Methodology and TUIs. Methodological Integration.

5.Case Study. Exploring OCH with TUIs. 
‚ÄĘ Places
‚ÄĘ Google Maps Timeline
‚ÄĘ Technology Specifications

 6. Future Work 
‚ÄĘ Design and Prototype
‚ÄĘ Technologic Specification
‚ÄĘ Evaluation

 7. Conclusion

8. References

Upgrade Process

I had my last meeting with my supervisors yesterday. At this stage I have managed to develop enough tools in the interface to be able to present the idea of my research.

The interface is a combination of interaction methods that include haptic and tangible interaction. The technology used in the interface includes computer vision and tabletop interaction based on Web technology such as JavaScript and Web Browser render.

The Upgrade Process

So I have this month to:

  • Define my contribution
  • Define methodology
  • Structure thesis

The thesis should include:

  • Problem / Question
  • Literature Review
  • Methodology
  • Discussion
  • Expected Results

I am currently doing a Web Science research. Web Science is an interdisciplinary research group. We are studying the Web from different perspectives. In the case of my research, I am trying to understand how can Cultural Heritage institutions can be enhance their engagement and pedagogic activities online. For this there is a wide range of disciplines that I have included in my research.

(Inter)Disciplinary approach

Human Computer Interaction

Through this approach I am intending to understand how people interact with the computer. Lets remember that on the Web, all interactions occur through an interface. For this reason it is important to study how these interactions might occur.

Human Information Interaction

Under this scope, I intend to produce a better understanding of how people/users relate, and process information. Usually when interfaces are developed designers focus merely on the interaction process without thinking beyond the tool itself. The HCI community has started adopting this process in their methodology but it has not yet been standardised as a design process in HCI.

Pedagogy / Psychology

Among many areas, I am focusing on Embodied Cognition. This thesis presents the idea that the mind is determined by the human body. This is to say that mental processes are not bounded to the mind itself. The body itself is a channel in which the human mind interacts and reasons about the world. This process is also highly linked to Constructivism, where people learn by experience. Constructivist learning experiences have commonly used cognitive processes as a pathway to empower learners. Even though constructivism is not limited to embodiment of activities, it might provide a positive pathway for pedagogic activities.

Enhancing Engagement with Online Museums

Museums as part of the Web are in need of producing meaningful pedagogic activities. The pedagogic element is essential to Cultural Heritage institutions. That is one of their main reasons of funding and even though organisations that are not in the education business can also benefit from such pedagogic activities. Nevertheless, this challenge is not easy to solve. For people to produce knowledge by themselves might not be as straightforward as one might think. There is a wide combination of how people might approach information spaces such as websites, databases, books, etc in order for them to extract knowledge.

By using the aforementioned disciplines in an interdisciplinary manner, I am attempting to produce an tangible user interface where people can be able to ask questions to a linked data system populated with cultural heritage data.