Microsoft Surface Tagged Objects and Tag Visualizations

Continued from Developing Microsoft Surface Applications

Microsoft Surface applications can recognize special tags besides fingers and  objects. These tags are similar to bar codes in concept and can store a particular value which can be retrieved by Surface’s vision system.

Tags are a pattern of white dots (infrared reflective) in black background (infrared absorbing) and are normally printed on a card or are printed and stuck to the plain surface of an object. Such objects with a tag are called as Tagged Objects.

Microsoft Surface supports two types of tags:


Byte Tags

• Stores 8 bits of data (1 byte).
• 256 possible unique values.
• Smaller size (3/4 x 3/4 inches).
• Reliable tracking even for faster moving tags.
• Represented in code by ByteTag structure.
• ByteTag.Value property represents the tag value.

Identity Tags

• Stores 128 bits of data (two 64 bit values).
• Larger range of possible unique values (i.e. 340,282,366,920,938,000,000,000,000,000,000,000,000).
• Larger size (1 x 1 inches).
• Functions better when tags are stationary or nearly stationary.
• Represented in code by IdentityTag structure.
• IdentityTag.Series and IdentityTag.Value together represents the tag value.

These tags are usually pre-printed and available from Microsoft or are printed using special tools that come with the Surface SDK.

Now let us build a sample application that deals with tagged objects. Create a new Surface project and add three Tag Visualization items (Add > New Item > Visual C# > Surface > v1.0 > Tag Visualization (WPF)). These items defines the UI that appears when a tagged object is placed on Surface. For now, let us just specify a height and width for it and then have a blank grid with a background color.

<s:TagVisualization x:Class="TagSample.BlueTags"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:s="http://schemas.microsoft.com/surface/2008"
    Loaded="BlueTags_Loaded" Height="50" Width="50">
    <Grid Background="Blue">
            
    </Grid>
</s:TagVisualization>

Now add the following XAML code in the SurfaceWindow1.xaml

<Grid Background="{StaticResource WindowBackground}" >
  <s:TagVisualizer VisualizationAdded="OnVisualizationAdded" >
    <s:TagVisualizer.Definitions>
      <!-- ByteTag: 10 = A -->
      <s:ByteTagVisualizationDefinition Value="10"
                                        Source="BlueTags.xaml">
      </s:ByteTagVisualizationDefinition>
      
      <!-- ByteTag: 250 = FA -->
      <s:ByteTagVisualizationDefinition Value="250"
                                        Source="RedTags.xaml">
      </s:ByteTagVisualizationDefinition>

      <!-- IdentityTag: 500 = 1F4, 1000 = 3E8 -->
      <s:IdentityTagVisualizationDefinition Series="500" Value="1000"
                                        Source="YellowTags.xaml">
      </s:IdentityTagVisualizationDefinition>
      
    </s:TagVisualizer.Definitions>
    <DockPanel LastChildFill="False">
      <TextBlock x:Name="uxDisplay"
                 HorizontalAlignment="Center"
                 DockPanel.Dock="Bottom"
                 Text="Watch here"/>
    </DockPanel>
  </s:TagVisualizer>
</Grid>

TagVisualizer is a content control that automatically displays visualization objects when a tag is placed on the control.  We keep this as the root containing control within Grid. Then we define three tags using TagVisualizationDefinition. Each definition specifies what kind of tag (Byte or Identity) we are using, its value, and the source file for TagVisualization. Finally we have a TextBlock to display the values.

Also note that we have defined VisualizationAdded event on the TagVisualizer. This gets fired whenever a tag is placed. Add the following code to handle this event.

void OnVisualizationAdded(object sender, TagVisualizerEventArgs e)
{
    String type = "";
    String value = "";

    if (e.TagVisualization.VisualizedTag.Type == TagType.Byte)
    {
        type = "Byte Tag";
        value = e.TagVisualization.VisualizedTag.Byte.Value.ToString();
    }
    else if (e.TagVisualization.VisualizedTag.Type == TagType.Identity)
    {
        type = "Identity Tag";
        value = e.TagVisualization.VisualizedTag.Identity.Series.ToString() + " - " + e.TagVisualization.VisualizedTag.Identity.Value.ToString();
    }

    uxDisplay.Text = type + ":" + value;
}

In this code, we are retrieving the tag information from the event arguments and then display it in a text box.



Run the application and try placing different tags. Note that the integer values need to be converted to Hexadecimal format while using in the simulator (I use Windows Calculator, change its View to Scientific, use ‘Dec’ option for decimal, type in a number and then choose ‘Hex’ option to convert it to Hexadecimal value) (E.g.: Value 10 in Hex = A).

Continued to: Microsoft Surface Applications - Deployment and Object Routing

Developing Microsoft Surface Applications

Continued from: Microsoft Surface Development Environment

Let us create a simple Surface Application to learn the development process.

Launch Visual Studio 2008 and create a new Surface project (File > New > Project > Visual C# > Surface > v1.0 > Surface Application (WPF))
The template will create all required files. Open the SurfaceWindow1.xaml file and following controls to it:

<DockPanel>
<s:SurfaceButton DockPanel.Dock="Bottom"
x:Name="uxButton"
Content="Click Me"
Height="50"
Width="100"
Click="uxButton_Click" />
<s:ScatterView x:Name="uxScatterView"
Background="LemonChiffon">
</s:ScatterView>
</DockPanel>

We have just added Surface version of the button and a new Surface specific control called ScatterView. The ScatterView control is the control that you should use when you have one or more UI elements that you want users to be able to move, rotate, or resize freely within a fixed area. It supports all these features without writing any extra code.

Now let us add the code to create ellipses and add them as ScatterView items in the button’s click handler:

private void uxButton_Click(object sender, RoutedEventArgs e)
{
Ellipse ell = new Ellipse();
ell.Height = 50;
ell.Width = 50;
ell.Fill = new SolidColorBrush(Colors.Red);

ScatterViewItem item = new ScatterViewItem();
item.Content = ell;
uxScatterView.Items.Add(item);
}

Now launch the Surface Simulator and run this application. The application will appear within the simulator. Keep clicking on the buttons to add items to the scatter view.


Note that the items get added to random positions. You can move and rotate the items with one finger. You can hold one finger to the edge of an item and use another finger to drag and then resize the item. Notice that the active item has a shadow effect and the shadow changes smoothly according to the items position. You can push the item to an edge with some speed and it will bounce back with smooth physics behavior.

All the Surface controls have contact events and this can be used to find the location of contact. Here is an example:

private void uxButton_ContactDown(object sender, ContactEventArgs e)
{
Point contactPosition = e.Contact.GetPosition(this);
}

For non-Surface controls, the contacts events can be attached like this:

<Canvas s:Contacts.ContactDown="Canvas_ContactDown" />

Using the combination of standard WPF controls and Surface special controls, it is possible to create multi-touch applications for Surface very easily.

Here are the list of Surface version of the standard controls:

• SurfaceWindow
• SurfaceButton
• SurfaceInkCanvas
• SurfaceSlider
• SurfaceScrollViewer
• SurfaceListBox
• SurfaceTextBox
• SurfacePasswordBox
• SurfaceMenu
• SurfaceContextMenu
• SurfaceCheckBox
• SurfaceRadioButton

And here is the list of Special Surface controls:

• ScatterView
• ScatterViewItem
• TagVisualizer
• TagVisualization

TagVisualizer helps in the use of tagged objects and can be used to show a visual when a tag is placed on the Surface unit.

Continued to Microsoft Surface Tagged Objects and Tag Visualizations

Microsoft Surface Development Environment

Continued from my previous post: Introduction to Microsoft Surface.

Surface Application Development Kit
Microsoft Surface applications are developed using WPF. You need Surface SDK Version 1.0 which is available only to some conference attendees and Microsoft partners. If you don’t have an actual Surface device, the SDK can be installed on a Windows computer; and the Surface Simulator from the SDK can be used to test the applications.

Here are the prerequisites for Surface SDK 1.0

• Windows Vista SP1 32 bit (Business/Enterprise/Ultimate)
• .NET Framework version 3.5 
• Microsoft XNA Framework Redistributable 2.0
• Microsoft DirectX End-User Runtime Web Installer
• Microsoft Visual Studio 2008

The SDK will install all required Surface libraries, Surface Simulator, Sample Applications, Documentation and Visual Studio project templates for Surface.

Surface User Experience
Before we get into the actual application development, let us see how Surface UX is different from conventional windows based applications. Even though the Surface application is running on top of Windows, the end user will not see any signs of it. Instead, the Surface device will be showing an attract application (similar to screensaver) which fills the complete display area. There will be four access points in the corners to open Launcher and explore available Surface applications. Note that only one application will be visible at a time. There is also an I’m done button to close all the applications and to return to the attract mode.


The above screenshot from the SDK documentation shows various parts of the Launcher (1. Application title | 2. Application description | 3. Application preview image | 4. Application icon image | 5. Access points | 6. I'm done button)

Surface Simulator
Surface Simulator is a Windows application with a fixed size (1224 × 868) used to simulate the actual Surface unit experience. It can be launched from ‘Start Menu > All Programs > Microsoft Surface SDK 1.0 > Tools > Surface Simulator’. Once the Surface Simulator is launched, any Surface applications launched there after will get displayed within the Simulator window.

Surface recognizes three types of contacts viz. Finger, Blob (any objects), and Tagged object (objects with a tag mark similar to barcode). There are two types of tags namely byte tags (up to 256 unique values) and identity tags (still in beta but supports more values).

All these contacts can be placed on the Simulator using the equivalent buttons as shown in the image below.


This screenshot shows the Data Visualizer sample application from SDK which is displaying the contact information.

To choose a contact on the application, the equivalent button is selected from the tool bar using mouse or its short key is pressed. The mouse pointer is then changed to the selected contact icon. For Tagged objects, the tag value can be specified in the adjacent text box.

To place a contact click and hold using the left mouse button, and to leave the contact there, click on the right moue button after that.


To remove a contact, choose the same contact selector, hover it on the placed contact, click left button and hold and then click right button of the mouse.

To rotate contact, use the mouse scroll wheel.

To select placed contacts, use the Contact Selector button. Drag a box around required contacts to select more than one. After selection, the contact can be moved around, rotated, and resized. To remove the selection click anywhere outside the contacts.

Below are the short cut keys available in Surface Simulator:

• Ctrl+F: Finger contact
• Ctrl+B: Blob contact
• Ctrl+T: Byte tag contact
• Ctrl+I: Identity tag contact
• Ctrl+S: Contact selector
• Ctrl+H: Hides all contacts
• Ctrl+S: Removes all contacts

The Surface Simulator also has a recording tab which can be used to record a set of actions (placing and movements of contacts) on the application. The recording is saved as a script file (*.scs) which is nothing but an xml file with key frames of contacts against time.  It is very useful to reproduce scenarios during testing of an application.

Continued to Developing Microsoft Surface Applications

Introduction to Microsoft Surface

Microsoft Surface is a multi-touch product from Microsoft which is developed as a software and hardware combination technology that allows user interaction with natural gestures and physical objects.
 

 It was announced on May 29, 2007 in US, and even now its availability is limited to companies in few countries like USA, UK, UAE, etc. Microsoft doesn’t say anything in their how to buy section about when they will release it to end users and to other countries like India.

This new surface computing platform opens up a a variety of possibilities for applications with Natural User Interfaces (NUI) as is evident from the following marketing video from Microsoft.





But its huge size and high price prevents many from appreciating this next generation device for digital interaction. See the following parody video which raises this exact point.



IdentityMine had ventured into the field of Surface application development from its earlier days itself and still continues to deliver products with remarkable user experiences.



Continued to Microsoft Surface Application Development Environment.

Upload Photos using Flickr API in .NET

You know there is Flickr, and you know they provide free API Service for accessing the online photos repository programmatically. Now let us see how easy it is to use those APIs in .NET.

In this example I am creating a .NET 2.0 Windows Forms Application (I used Visual Studio 2008) in C# to upload a photo to your Flickr account and then create a new set (album) and assign that photo to this set.

Here are the steps to do this:

• Get the API Key from Flickr (This key and its secret code are required to access the service)
• Go to Your API Keys section on Flickr.
• Click on ‘Apply for a new key online’ link.
• Complete the process. I used the Non-Commercial option to get the key immediately. You will have to provide the details of your application. Once this is done, click on Edit key details and update all the information.
• Note down the Key (it will look like 3e1f3c83c5e92ead353be33162fr4883) and Shared Secret (it will look like 700f4d5220833d34)
• Get the FlickrNet API Library (This is a .NET wrapper class library around Flickr API and makes it very easy to call the services)
• Go to http://flickrnet.codeplex.com/
• Download either just the binaries or the source code if you would like to debug into the library.
• FlickrNet.dll is the only one file you will require.
• Design the application interface
• I created a WinForm application as shown below with options to authenticate the user and to choose a picture file for uploading.

Now let us understand some theories behind Flickr API before we get into coding. Skip this if all you bother is to get your code working. :)

You need to do an authentication to the flickr account you are trying to access. This is a one time process and is done by directing the user to a URL (generated using the service) which will then open up in a browser. This process is initiated using ‘Start Auth’ button. Once the user logs into the flickr account and approve the access request from this application, he or she will have to come back to this application and click on ‘Complete Auth’. In this method, we get a ‘Token’ which uniquely identifies the user. This token can be saved and used again to access this user’s account without requiring further authentication process.

• Add code to call Flickr API methods

First thing is to add a reference to the FlickrNet (either direct dll or the project reference if you use source code) in your project. Next declare the required variables.

private string flickrFrob = "";
private long fileSize = 1;
FlickrNet.Flickr myFlickr;

Then in the Form’s load event, provide your API Key and Share Secret codes and create the Flickr object. Event handlers are also added here.

private void MainForm_Load(object sender, EventArgs e)
{
    uxKey.Text = "3e1f3c83c5e92ead353be33162fr4883";
    uxSecret.Text = "700f4d5220833d34";
    myFlickr = new FlickrNet.Flickr(uxKey.Text, uxSecret.Text);
    uxOpen.Click += new EventHandler(uxOpen_Click);
    uxAuthStart.Click += new EventHandler(uxAuthStart_Click);
    uxAuthComplete.Click += new EventHandler(uxAuthComplete_Click);
    uxUpload.Click+=new EventHandler(uxUpload_Click);
    myFlickr.OnUploadProgress += new FlickrNet.Flickr.UploadProgressHandler(myFlickr_OnUploadProgress);
}

Now let us open an image for uploading…

void uxOpen_Click(object sender, EventArgs e)
{
    OpenFileDialog ofd = new OpenFileDialog();
    ofd.Filter = "Pictures|*.jpg";
    if (ofd.ShowDialog() == DialogResult.OK)
    {
        uxFile.Text = ofd.FileName;
        fileSize = ofd.OpenFile().Length;
        Image img = Image.FromFile(ofd.FileName);
        uxPicture.Image = img;
    }
}

and then start authentication.

void uxAuthStart_Click(object sender, EventArgs e)
{
    flickrFrob = myFlickr.AuthGetFrob();
    string url = myFlickr.AuthCalcUrl(flickrFrob, FlickrNet.AuthLevel.Write);
    System.Diagnostics.Process.Start(url);
}

This uses something called Frob (cookie kinda stuff) and generates a URL which is then opened up in the user’s default browser. Remember, at this point, user will have to sign in and authenticate the access request from this application.

Once the user completes the authentication and comes back to the application, we gets the token for that user.

void uxAuthComplete_Click(object sender, EventArgs e)
{
    try
    {
        FlickrNet.Auth auth = myFlickr.AuthGetToken(flickrFrob);
        myFlickr.AuthToken = auth.Token;
        uxToken.Text = auth.Token;
        uxUser.Text = auth.User.UserId;
    }
    catch (FlickrNet.FlickrException ex)
    {
        MessageBox.Show("User didn't approve!\n\n" + ex.Message);
    }
}

The token is assigned to the Flickr object to enable access to that user’s account. It can be saved for later use without doing this authentication process again.

Now we are all set to upload the picture to flickr.

private void uxUpload_Click(object sender, EventArgs e)
{
    uxProgress.Maximum = Convert.ToInt32(fileSize / 1000);
    uxProgress.Value = 0;    try
    {
        FlickrNet.Photo p = new FlickrNet.Photo();
        System.Threading.Thread threadUpload = new System.Threading.Thread(delegate()
        {
            p.PhotoId = myFlickr.UploadPicture(uxFile.Text, "Sample Title", "Sample Description", "Sample Tag", false, false, true);
            p.UserId = uxUser.Text;
            FlickrNet.Photoset mySet = myFlickr.PhotosetsCreate("New Photoset", p.PhotoId);
            System.Diagnostics.Process.Start(p.WebUrl);
        });        threadUpload.Start();
    }
    catch (Exception ex)
    {
        MessageBox.Show("Upload failed!\n\n" + ex.Message);
    }
}

Here we set the progress bar values and then use a new thread different from UI thread to call the Upload method. We also create a new photo set and add this photo to it.

The last thing is to make sure that the progress bar is updated properly.

private void myFlickr_OnUploadProgress(object sender, FlickrNet.UploadProgressEventArgs e)
{
    if (e.UploadComplete)
    {
        this.Invoke((MethodInvoker)delegate()
        {
            uxProgress.Value = 0;
        });
    }
    else
    {
        this.Invoke((MethodInvoker)delegate()
        {
            uxProgress.Value = e.Bytes / 1000;
        });
    }
}

That’s it… Hope this was useful to you and as always, feel free to ask any questions you may have on this…