Tangible Interaction with Sifteo Cubes.

 Dynamic Fiducial Pyfos (DFPs)

During the last weeks I have started developing what I would like to call DFPs short for dynamic-fiducial-pyfos. With the help of some friends I have now a basic skeleton to extend my interactive experiments. Here is a video of the result:

 

Tangible Interaction and Pyfos

After I submitted my upgrade draft, I realised that I was going to encounter some issues when working with pyfos when using them as part of the fiducials for the TUIO system. Since users have to combine different concepts (e.g. Roman + pottery or painter + 1800) this will result in a numerous amount of pyfos. The interface already has several objects that can not be removed since they are part of the basic interactions such as: map navigation, box dragging, etc… Therefore, I decided to explore a little bit further. I need to find a way to extend the capabilities of pyfos.

Pyfos have three main states: token, constraint and token+constraint. The TAC (Token and Constraints) paradigm in Tangible User Interfaces (TUIs) offers a set of constructs of how these objects react. Nevertheless, it can be argued that technology can offer pyfos that can self adapt or expand the constraints that bound them. This due to the fact that physical objects such as pyfos cannot morph or change according to user needs.

TUIO Javs

Other researchers are exploring how these TAC approaches can be expanded. It is this search that directed me to explore with mini-displays and sensors. Since I had worked with some Internet of Things and Arduino, I thought of designing some display that detected two different RFIDs to make the combinations and display a specific result. That result display presents the final combination in a form of fiducial so the TUIO interface can detect it. This way, users can pre-design a concept combination thus integrating it in a final dynamic-fiducial-pyfo (DFP) that encompasses that prior combination. Most importantly, DFPs can also produce other display results on the table-top without clustering tools or options.

02

Producing DFPs

There are some alternatives to produce DFPs out there. Many of them require to be built from scratch, but there are some products that can be adapted to our needs without having to fiddle or hack any of the electronics. Although there is a wide variety of tools out there, these are the most ‘approachable’ tools that I encountered.

Educubes

Since I had worked with Arduino and Internet of Things before, I thought of building a mini-display with either RFID or any other type of sensor. I found the Educubes project that presented a good opportunity to start developing for this idea. This might also prove to be beneficial since there are some TFT mini-displays that support touchscreen actions as well.

Educubes by Adafruit
Educubes by Adafruit

 Although this presented a good opportunity to develop, I needed to start producing tools that could work straight away instead of focusing on the electronics. Moreover I thought that the size of the electronics is still quite big for them to be used on the prototype. But it was mainly the issue of working with a wide range of electronics and hacking them so they can do what I wanted.

Sifteo Cubes

After searching I encountered Sifteo Cubes. These cubes already provide a very nice presentation that encompasses a wide range of electronics and a mini-display. Moreover these cubes can be programmed through an SDK provided by the same company. I decided to jump ahead and ordered a second generation Sifteo Cubes.

My surprise was at the moment of using the SDK. I was not the first user to be put down by its complexity. The Sifteo SDK works with C++ with other command tools to run installations and device management. Moreover, through the forums I encountered that the released SDK contained some bugs, which made some of the tutorials not to work.

Nevertheless, I encountered some compiled SDK in GitHub such as Investio and Sifteo Blickets. They provided me with some hints into how to actually start using and managing the cubes. I still had to learn how to program what I needed. So I started first learning how to make interactions. There is a base of few interactions that are supported by the sensors in the cube that include: tilt, pair, shake and press. 

Sifteo Actions
Sifteo Actions

Although it seems quite nice in pictures, the task was not so simple. Since I do not come from a programming background, working with C++ was a huge challenge. First I did a tutorial on neighbouring. Here is the video:

A few days after, I started working with other actions such as tilt and press. It is relevant to mention that I worked using the examples that came with the SDK so, the interactions were pretty much pre-designed and I was just learning a few basic commands that might be used. Here is a video of this stage.

The problem started when working with my specific requirements. I needed an array of options per cube that could be combined between them. This so the final combinations could be applied in the Europeana TUIO system. Using C++ this was not so straightforward. In a nutshell, this is what I needed to create:

Basic combination skeleton
Basic combination skeleton

It took me a lot of time and effort to find a way to program this interaction. I could program something like this with other languages but not with C++. Therefore I asked for some help to develop this. Kevin Lesur from One Life Remains gave me a hand with this. So this is was the basic skeleton built for the interactions:

Operations and interactions skeleton.
Operations and interactions skeleton.

Two cubes are required to make the combination through neighbouring. When they are combined a third cube presents its combination that will eventually show a fiducial making it a DFP. To navigate between the cubes options, users can tilt the cubes in either direction.

This way I am hoping to now carry on and go back to the TUIO experiments and see how these DFPs work on the tabletop system.

 

June 17 – Visualising // Managing Results

Visualising and Dealing with the Results.

It has been about three weeks since my last post. This is because I have been focusing more on specialising in using JavaScript and more specifically JQuery. JQuery has the potential of making it easier to exploit JavaScript on Web content.

Updates on the Interface

Displaying Results

When users make queries, they will be retrieving vast sets of information. This will prove to be a complex task. For this reason, I need to find a way on how they can be able to sort such content. One of the ways that I have envisioned is by providing an area where results can be sorted and then manipulated by assigning them to a fiducial item. Users should be able to offload that information thus simplifying the process.

dropUI

The drop box will serve the function of holding the result that the user wants to select. When he or she puts it back into the original results panel, the item will be highlighted in a different colour, otherwise the user can clear the result and remove the item from the interface. By highlighting the returned item, users can be able to identify the ones that they found particularly relevant.

dropUIhighlight

The returned element has a heart icon that shows that the element was kept from the drop box. The box on the other hand just contains a lightbulb showing that it is highlighted. Moreover, I have also added clear buttons so users can be able to remove unselected elements from the specific panel.

I should be able to create a fiducial  with a screen, or perhaps a frame where the visual can be contained. When the user drags the frame, the drop box moves along with it. The physical object should allow users to literally grasp specific datasets so they can be compared or used for further queries.

dropUIfidJQuery and JQuery UI

There is a huge potential that JQuery has to offer to these type of interfaces. The challenge will be on transforming common graphic user interface tools into the physical tools. My previous post indicates some of the many possibilities of making the queries and navigating once the result has been made. This drop box should be able to help the querying process along with the data navigation.

Here is a video of the table top system still using TUIO as the interactivity connection.

Tangible User Interface for Europeana’s API

I have been working with TUIO for the main interface in between Europeana’s API and a user. One of my main research questions is how to can users make questions through tangible interfaces. In this case, users should be able to make questions about cultural heritage content from different organisations. In this case users can be to access data from over 30 million metadata records that include books, photos, art, audio and artefacts among others. On one hand having access to all this content may benefit users since they will have vast sets of information to answer their questions. On the other they might get lost with all the data that is available for them.

When working with vast sets of information, users can benefit by dissecting specific items of the information that they are looking for. Nevertheless, this process might prove difficult might require a lot of concentration. By offloading this mental process onto physical objects, users might pace their thinking process and assist themselves by using the physical objects as an aid.

My intention is to produce such objects that can help users solving questions and finding information from a data portal such as Europeana. For this I have created a starting skeleton of essential objects that can then be transformed to queries on the API.

Essential Queries

Who – What

Some of the most essential queries might include who or what are looking for. These will be represented individually.

who and what icons

query = who:"string"
query = what:"string"

Places

Geolocation

By positioning the fiducials on top of a map, users can be able to add geoCoordinates on to the query. This can be used by adding two values either by tapping or by using two fiducials.

pos

qf=pl_wgs84_pos_lat:3A%5B01+TO+02%5D & qf=pl_wgs84_pos_long:3A%5B03+TO+04%5D

Country

Countries are different to geoLocation. Countries are defined by name that are part of a human perception and not a geographic one. This way this fiducial will detect it’s x and y position and add the name of the closest place to the query. There will be several of these fiducials for adding more places to the query.

country

query = COUNTRY:string

Time

Another value that can be added to the query are the dates for the time period that you want to constrain the data to. Time being abstract, its very difficult to represent and manipulate. Europeana provides a starting year and an ending year. For this reason this will be using two ‘dials’ to forward or rewind starting or ending year.

year

query = YEAR: 0000 TO 0000

Boolean Operations

Basic boolean operations can be added to the query to produce a more specific question. Operations such as AND, NOT and OR can be used. This fiducial has to be used in addition of the fiducials previously mentioned otherwise there will be no values to compare to.

This marker commonly will require one of the three options. Nevertheless, users might benefit by looking at all the options to analyse what they are asking.

boolean

query  = where:(Paris NOT France)

Container

To put everything together, users might benefit of structuring their thoughts on a template where they can finalise their sentence or question. This does not mean that they must place artefacts in such order but is merely a starting point to give an approach of how they might structure questions.

console

There is still a wide variety of query options that can be added, but this is the essential information that users will require to input at a specific time in order to retrieve meaningful results.

Google Maps API

Google Maps API

geomap

The Google Maps API allows users to populate data on maps powered by Google. Google provides an API where a wide variety of settings and technology implementations can be used.

This certainly brings a huge potential when using museum and archaeology data. For example Europeana provides latitude and longitude coordinates as part of their data model. In order for me to use this, I started playing with some visualisation tools in gMaps.

Styling the Map

The first thing I had to learn was to apply styles to the map. This is quite straightforward, especially using a styler assistant such as the Google Maps Style Wizard. This will generate the JSon  so you can just apply it on your map. Moreover, there are repositories where the community deposits their own designs and colour schemes so you can just implement them on your map. Snazzy Maps is a good place to start.

Generate Icons

I also had to learn how to place markers and icons on the map. For this, instead of using the classic gMaps marker, I generated my own set of 8bit icons. I used Illustrator and exported them as .png format.

gmapsmarkers

Use Some GeoData

Finally I started using some geoData and placing the markers in the map. For this I used a map that contained information about textile designers of indigenous Mayan communities from Chiapas. I created arrays of info and coordinates according to their ethnic group.

geoarray

Each one of the arrays contain each ethnic group. The rest was just repeating the same process until I had them all organised into colour-groups.

** Check the map in my website **

Next step

The next step is to “auto-populate” the gMaps coordinates automatically from the Europeana database.