Microsoft Surface 2.0: the multi-touch wonder

It was worth the wait and the game has completely changed with the release of Microsoft Surface 2.0, the latest version of Microsoft's multi-touch product.

Now Microsoft has a partner in Samsung who will help with the hardware. The old bulky Surface table is now replaced with a sleek LCD panel. The new product, named as "Samsung SUR40 for Microsoft Surface", is similar to a large LCD TV but with the multi-touch capability. People have already started calling it the world's biggest iPad.

It uses a new technology from Samsung, PixelSense, which gives LCD panels the power to see without the use of cameras.  The tiny infrared sensors wedged between pixels can sense touches, objects and tags.

Here are some specs of Surface 2.0 with the specs of Surface 1.0 in brackets:

• Device Display: 40-inch LCD panel with Gorilla Glass (30-inch reflective surface with acrylic)
• Resolution: 1920x1080, full HD 1080p, with a 16:9 aspect ratio (1024 x 768, no HD support)
• Multi-touch technology: PixelSense (Projector and 5-cameras system)
• Multi-touch capability: 50 simultaneous touches (52 simultaneous touches)
• Device Form: 40-inch diagonal panel with 4-inch thickness (30-inch display in a table-like form factor, 22 inches high, 21 inches deep, and 42 inches wide)
• Weight: 39.5 Kg (68 Kg)
• Mounting: Horizontal and Vertical (Only Horizontal)
• Processor: AMD Athlon™ II X2 Dual-Core Processor 2.9GHz paired with the AMD Radeon HD 6700M Series GPU featuring DirectX 11 support (Intel Core 2 Duo @ 2.13 GHz)
• RAM: 4 GB DDR3 (2 GB DDR2)
• Hard Disk: 320 GB/7200 RPM (250GB SATA Hard Drive)
• HDMI In / HDMI Out: Yes (No)
• Operating System: Windows 7 Professional 64-bit (Windows Vista Professional 32-bit)
• Development Platform: Surface 2.0 SDK / WPF 4.0 / .NET 4.0 (Surface 1.0 SP1 SDK  / WPF 3.5 / .NET 3.5)
• Cost: $7,600 ($12,500) -- Approximate Values

I had enjoyed developing applications for Surface 1.0 and was always excited to explain about them to others ("You don't use a mouse or keyboard... you just drag things with your fingers like you see in Avatar!"). But it never took off as expected and was not seen in common use, mostly due to its bulky size. Now with Surface 2.0, I'm sure things will be a lot different.

ScatterView for Windows/WPF is here!

Finally, the wait is over and Microsoft has released Surface Toolkit for Windows Touch. Now we will be able to create Windows Applications that are touch enabled and works on Windows 7 computers with touch screens. The Toolkit brings many good old Surface touch controls including the much awaited ScatterView to the Windows platform. The good thing is that tremendous research has been done by Microsoft team in the areas of user experience and usability for these controls and hence we developers can save precious time otherwise spent on developing custom touch controls.

The controls included are:
LibraryBar, LibraryContainer, LibraryStack, ScatterView, SurfaceButton, SurfaceCheckBox, SurfaceInkCanvas, SurfaceListBox, SurfaceRadioButton, SurfaceScrollViewer, SurfaceSlider, SurfaceWindow.

These touch specific controls responds to touches, stylus and mouse clicks and have many special touch features (One example is the bulging out of scroll bars when touched).

One thing that is missing from the Toolkit is a Simulator similar to the Surface Simulator to help out those developers who do not have a touch screen.  Also the applications written using these controls will now only work on Windows 7 and not on Surface. Hopefully these may change when they release the next version of Surface (Surface 2.0) based on .NET 4.0.

Microsoft Surface Toolkit for Windows Touch Beta can be downloaded from here and more details can be found at the Surface Blog.

Microsoft Surface Applications - Deployment and Object Routing

Continued from: Microsoft Surface Tagged Objects and Tag Visualizations

When users interact with a Microsoft Surface unit, they launch the applications from Launcher.  Launcher is the menu that displays the applications that are available to users in a horizontal interactive strip.


 As part of deploying the Surface application, we must register it with Surface so that it is available in the Launcher. This is done by copying the application’s xml file to the Surface’s program data folder (%PROGRAMDATA%\Microsoft\Surface\Programs).

This application xml file is created by default when the Visual Studio project template is used. Otherwise, we can create a normal xml file with the file name equivalent as the project (executable) name and with the following tags.

<?xml version="1.0" encoding="utf-8" ?>
<ss:ApplicationInfo
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:ss="http://schemas.microsoft.com/Surface/2007/ApplicationMetadata">
  <Application>
    <Title>My Surface App</Title>
    <Description>This is a sample Surface application.</Description>
    <ExecutableFile>%ProgramFiles%\MyCompany\MyApp\MyApp.exe</ExecutableFile>
    <Arguments></Arguments>
    <IconImageFile>%ProgramFiles%\MyCompany\MyApp\Resources\icon.png</IconImageFile>
    <Preview>
      <PreviewImageFile>%ProgramFiles%\MyCompany\MyApp\Resources\iconPreview.png</PreviewImageFile>
    </Preview>
    <Tags>
      <ByteTag Value="C0">
        <Actions>
          <Launch />
        </Actions>
      </ByteTag>
    </Tags>
  </Application>
</ss:ApplicationInfo>

The <Application> tag specifies that this is a normal Surface application. Other options are <AttractApplication> for attract mode (similar to screen saver) applications and <ServiceApplication> for apps that run in the background and does not have a UI.

The <Title> is displayed on the Launcher all the time, and <Description> is displayed when the application is selected from the Launcher. <ExecutableFile> specifies the path to the actual application exe. If the application is accepting any command line arguments it can be supplied through <Arguments> tag. <IconImageFile> specifies the png image to be displayed in the Launcher as application icon and <Preview> specifies the image to be displayed when the application is selected in the Launcher. This can be a bigger static image(<PreviewImageFile>) or a slideshow of images(<SlideshowImageFile>) or a movie (<MovieFile>).

The <Tags> specify the tags used by the application and <Launch> is used to specify that the particular tag value is used for object routing (which is discussed later).

To summarize the deployment steps again:

• Build your Surface application in Release mode.
• Copy all the files from the build folder to any location in the Surface computer (typically %ProgramFiles%\CompanyName\ApplicationName\).
• Make sure that this path is specified in the Application’s xml file.
• Copy the Application’s xml file to (%PROGRAMDATA%\Microsoft\Surface\Programs folder.

That’s all to it and the application will now be available in the Launcher for the user to see and launch.

Tagged Object Routing
On 2009-May-10, Microsoft released its first Service Pack (SP1) to Surface 1.0 with lots of new features and improvements. Read about them at official Surface blog.

Object Routing is one of the new SP1 features that enable the developer to configure tagged objects which can open applications without using Launcher. The tagged object can be placed anywhere on the Surface interface (attract applications, other non-registered applications or the Launcher) to bring up the list of applications that are registered with that tag. The applications will appear as tiny bubbly icons that smoothly pops up from the object and can be launched by touching them.

 
To enable the object routing for an applications:

• Specify the required type of tag (Byte/Identity) within the <Tags> section of the application’s xml file and specify its value and have the <Actions> as <Launch>.

<ByteTag Value="C0">
  <Actions>
    <Launch />
  </Actions>
</ByteTag>

or

<IdentityTag Series="4245A8E4901C2C0B">
  <Actions>
    <Launch /> 
  </Actions>
</IdentityTag>

I have used an example value of C0 for Byte tag and 4245A8E4901C2C0B for Identity tag.

• Create a new registry key using this tag value under TagInfo section as below:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\ByteTags\C0
or

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\IdentityTags\4245A8E4901C2C0B

• Add the following String Value items on this key
• Text : ‘object routing card title’.
• BoundingBoxWidth : ‘3.25’.
• BoundingBoxHeight : ‘2.125’.
• PhysicalCenterOffsetFromTagX : ‘0’.
• PhysicalCenterOffsetFromTagY : ‘0’.
• OrientationOffsetFromTag : ‘0’.

This specifies the size of tagged object (business card size is 3.25 x 2.125 inches) and the location of the tag within this object (Zero means there is no offset from the center of the object; the tag is exactly on the center of this business card sized object)

Note that the other registry settings (enabled by default) to enable Object Routing (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\v1.0\Shell\EnableObjectRouting ) must be set as well.

That’s all the settings we need to do to get object routing work. Now if you place a tag with the specified value, all the applications that have this tag value in their xml file registered will popup from this object.

If you have done everything right, and still its not working as expected, refer the event log (run the command eventvwr) and navigate to Applications and Services Logs/Microsoft/Surface/Shell/Operational for any possible warnings or errors. Chances are that there is some error in the application's xml file or the registry settings. A sample warning (when the required registry entry is not present) looks like below:

The application info file C:\ProgramData\Microsoft\Surface\Programs\Macroni.xml refers to tag 29EB049F73061803, however, some configuration information for 29EB049F73061803 is missing from the registry. The default value '2.125' will be used for the missing setting.
Path to the tag's missing configuration setting in the registry:
HKEY_LOCAL_MACHINE\Software\Microsoft\Surface\TagInfo\v1.0\IdentityTags\29EB049F73061803\BoundingBoxHeight

Happy coding!

Microsoft Surface Tagged Objects and Tag Visualizations

Continued from Developing Microsoft Surface Applications

Microsoft Surface applications can recognize special tags besides fingers and  objects. These tags are similar to bar codes in concept and can store a particular value which can be retrieved by Surface’s vision system.

Tags are a pattern of white dots (infrared reflective) in black background (infrared absorbing) and are normally printed on a card or are printed and stuck to the plain surface of an object. Such objects with a tag are called as Tagged Objects.

Microsoft Surface supports two types of tags:


Byte Tags

• Stores 8 bits of data (1 byte).
• 256 possible unique values.
• Smaller size (3/4 x 3/4 inches).
• Reliable tracking even for faster moving tags.
• Represented in code by ByteTag structure.
• ByteTag.Value property represents the tag value.

Identity Tags

• Stores 128 bits of data (two 64 bit values).
• Larger range of possible unique values (i.e. 340,282,366,920,938,000,000,000,000,000,000,000,000).
• Larger size (1 x 1 inches).
• Functions better when tags are stationary or nearly stationary.
• Represented in code by IdentityTag structure.
• IdentityTag.Series and IdentityTag.Value together represents the tag value.

These tags are usually pre-printed and available from Microsoft or are printed using special tools that come with the Surface SDK.

Now let us build a sample application that deals with tagged objects. Create a new Surface project and add three Tag Visualization items (Add > New Item > Visual C# > Surface > v1.0 > Tag Visualization (WPF)). These items defines the UI that appears when a tagged object is placed on Surface. For now, let us just specify a height and width for it and then have a blank grid with a background color.

<s:TagVisualization x:Class="TagSample.BlueTags"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:s="http://schemas.microsoft.com/surface/2008"
    Loaded="BlueTags_Loaded" Height="50" Width="50">
    <Grid Background="Blue">
            
    </Grid>
</s:TagVisualization>

Now add the following XAML code in the SurfaceWindow1.xaml

<Grid Background="{StaticResource WindowBackground}" >
  <s:TagVisualizer VisualizationAdded="OnVisualizationAdded" >
    <s:TagVisualizer.Definitions>
      <!-- ByteTag: 10 = A -->
      <s:ByteTagVisualizationDefinition Value="10"
                                        Source="BlueTags.xaml">
      </s:ByteTagVisualizationDefinition>
      
      <!-- ByteTag: 250 = FA -->
      <s:ByteTagVisualizationDefinition Value="250"
                                        Source="RedTags.xaml">
      </s:ByteTagVisualizationDefinition>

      <!-- IdentityTag: 500 = 1F4, 1000 = 3E8 -->
      <s:IdentityTagVisualizationDefinition Series="500" Value="1000"
                                        Source="YellowTags.xaml">
      </s:IdentityTagVisualizationDefinition>
      
    </s:TagVisualizer.Definitions>
    <DockPanel LastChildFill="False">
      <TextBlock x:Name="uxDisplay"
                 HorizontalAlignment="Center"
                 DockPanel.Dock="Bottom"
                 Text="Watch here"/>
    </DockPanel>
  </s:TagVisualizer>
</Grid>

TagVisualizer is a content control that automatically displays visualization objects when a tag is placed on the control.  We keep this as the root containing control within Grid. Then we define three tags using TagVisualizationDefinition. Each definition specifies what kind of tag (Byte or Identity) we are using, its value, and the source file for TagVisualization. Finally we have a TextBlock to display the values.

Also note that we have defined VisualizationAdded event on the TagVisualizer. This gets fired whenever a tag is placed. Add the following code to handle this event.

void OnVisualizationAdded(object sender, TagVisualizerEventArgs e)
{
    String type = "";
    String value = "";

    if (e.TagVisualization.VisualizedTag.Type == TagType.Byte)
    {
        type = "Byte Tag";
        value = e.TagVisualization.VisualizedTag.Byte.Value.ToString();
    }
    else if (e.TagVisualization.VisualizedTag.Type == TagType.Identity)
    {
        type = "Identity Tag";
        value = e.TagVisualization.VisualizedTag.Identity.Series.ToString() + " - " + e.TagVisualization.VisualizedTag.Identity.Value.ToString();
    }

    uxDisplay.Text = type + ":" + value;
}

In this code, we are retrieving the tag information from the event arguments and then display it in a text box.



Run the application and try placing different tags. Note that the integer values need to be converted to Hexadecimal format while using in the simulator (I use Windows Calculator, change its View to Scientific, use ‘Dec’ option for decimal, type in a number and then choose ‘Hex’ option to convert it to Hexadecimal value) (E.g.: Value 10 in Hex = A).

Continued to: Microsoft Surface Applications - Deployment and Object Routing

Developing Microsoft Surface Applications

Continued from: Microsoft Surface Development Environment

Let us create a simple Surface Application to learn the development process.

Launch Visual Studio 2008 and create a new Surface project (File > New > Project > Visual C# > Surface > v1.0 > Surface Application (WPF))
The template will create all required files. Open the SurfaceWindow1.xaml file and following controls to it:

<DockPanel>
<s:SurfaceButton DockPanel.Dock="Bottom"
x:Name="uxButton"
Content="Click Me"
Height="50"
Width="100"
Click="uxButton_Click" />
<s:ScatterView x:Name="uxScatterView"
Background="LemonChiffon">
</s:ScatterView>
</DockPanel>

We have just added Surface version of the button and a new Surface specific control called ScatterView. The ScatterView control is the control that you should use when you have one or more UI elements that you want users to be able to move, rotate, or resize freely within a fixed area. It supports all these features without writing any extra code.

Now let us add the code to create ellipses and add them as ScatterView items in the button’s click handler:

private void uxButton_Click(object sender, RoutedEventArgs e)
{
Ellipse ell = new Ellipse();
ell.Height = 50;
ell.Width = 50;
ell.Fill = new SolidColorBrush(Colors.Red);

ScatterViewItem item = new ScatterViewItem();
item.Content = ell;
uxScatterView.Items.Add(item);
}

Now launch the Surface Simulator and run this application. The application will appear within the simulator. Keep clicking on the buttons to add items to the scatter view.


Note that the items get added to random positions. You can move and rotate the items with one finger. You can hold one finger to the edge of an item and use another finger to drag and then resize the item. Notice that the active item has a shadow effect and the shadow changes smoothly according to the items position. You can push the item to an edge with some speed and it will bounce back with smooth physics behavior.

All the Surface controls have contact events and this can be used to find the location of contact. Here is an example:

private void uxButton_ContactDown(object sender, ContactEventArgs e)
{
Point contactPosition = e.Contact.GetPosition(this);
}

For non-Surface controls, the contacts events can be attached like this:

<Canvas s:Contacts.ContactDown="Canvas_ContactDown" />

Using the combination of standard WPF controls and Surface special controls, it is possible to create multi-touch applications for Surface very easily.

Here are the list of Surface version of the standard controls:

• SurfaceWindow
• SurfaceButton
• SurfaceInkCanvas
• SurfaceSlider
• SurfaceScrollViewer
• SurfaceListBox
• SurfaceTextBox
• SurfacePasswordBox
• SurfaceMenu
• SurfaceContextMenu
• SurfaceCheckBox
• SurfaceRadioButton

And here is the list of Special Surface controls:

• ScatterView
• ScatterViewItem
• TagVisualizer
• TagVisualization

TagVisualizer helps in the use of tagged objects and can be used to show a visual when a tag is placed on the Surface unit.

Continued to Microsoft Surface Tagged Objects and Tag Visualizations

Microsoft Surface Development Environment

Continued from my previous post: Introduction to Microsoft Surface.

Surface Application Development Kit
Microsoft Surface applications are developed using WPF. You need Surface SDK Version 1.0 which is available only to some conference attendees and Microsoft partners. If you don’t have an actual Surface device, the SDK can be installed on a Windows computer; and the Surface Simulator from the SDK can be used to test the applications.

Here are the prerequisites for Surface SDK 1.0

• Windows Vista SP1 32 bit (Business/Enterprise/Ultimate)
• .NET Framework version 3.5 
• Microsoft XNA Framework Redistributable 2.0
• Microsoft DirectX End-User Runtime Web Installer
• Microsoft Visual Studio 2008

The SDK will install all required Surface libraries, Surface Simulator, Sample Applications, Documentation and Visual Studio project templates for Surface.

Surface User Experience
Before we get into the actual application development, let us see how Surface UX is different from conventional windows based applications. Even though the Surface application is running on top of Windows, the end user will not see any signs of it. Instead, the Surface device will be showing an attract application (similar to screensaver) which fills the complete display area. There will be four access points in the corners to open Launcher and explore available Surface applications. Note that only one application will be visible at a time. There is also an I’m done button to close all the applications and to return to the attract mode.


The above screenshot from the SDK documentation shows various parts of the Launcher (1. Application title | 2. Application description | 3. Application preview image | 4. Application icon image | 5. Access points | 6. I'm done button)

Surface Simulator
Surface Simulator is a Windows application with a fixed size (1224 × 868) used to simulate the actual Surface unit experience. It can be launched from ‘Start Menu > All Programs > Microsoft Surface SDK 1.0 > Tools > Surface Simulator’. Once the Surface Simulator is launched, any Surface applications launched there after will get displayed within the Simulator window.

Surface recognizes three types of contacts viz. Finger, Blob (any objects), and Tagged object (objects with a tag mark similar to barcode). There are two types of tags namely byte tags (up to 256 unique values) and identity tags (still in beta but supports more values).

All these contacts can be placed on the Simulator using the equivalent buttons as shown in the image below.


This screenshot shows the Data Visualizer sample application from SDK which is displaying the contact information.

To choose a contact on the application, the equivalent button is selected from the tool bar using mouse or its short key is pressed. The mouse pointer is then changed to the selected contact icon. For Tagged objects, the tag value can be specified in the adjacent text box.

To place a contact click and hold using the left mouse button, and to leave the contact there, click on the right moue button after that.


To remove a contact, choose the same contact selector, hover it on the placed contact, click left button and hold and then click right button of the mouse.

To rotate contact, use the mouse scroll wheel.

To select placed contacts, use the Contact Selector button. Drag a box around required contacts to select more than one. After selection, the contact can be moved around, rotated, and resized. To remove the selection click anywhere outside the contacts.

Below are the short cut keys available in Surface Simulator:

• Ctrl+F: Finger contact
• Ctrl+B: Blob contact
• Ctrl+T: Byte tag contact
• Ctrl+I: Identity tag contact
• Ctrl+S: Contact selector
• Ctrl+H: Hides all contacts
• Ctrl+S: Removes all contacts

The Surface Simulator also has a recording tab which can be used to record a set of actions (placing and movements of contacts) on the application. The recording is saved as a script file (*.scs) which is nothing but an xml file with key frames of contacts against time.  It is very useful to reproduce scenarios during testing of an application.

Continued to Developing Microsoft Surface Applications

Introduction to Microsoft Surface

Microsoft Surface is a multi-touch product from Microsoft which is developed as a software and hardware combination technology that allows user interaction with natural gestures and physical objects.
 

 It was announced on May 29, 2007 in US, and even now its availability is limited to companies in few countries like USA, UK, UAE, etc. Microsoft doesn’t say anything in their how to buy section about when they will release it to end users and to other countries like India.

This new surface computing platform opens up a a variety of possibilities for applications with Natural User Interfaces (NUI) as is evident from the following marketing video from Microsoft.





But its huge size and high price prevents many from appreciating this next generation device for digital interaction. See the following parody video which raises this exact point.



IdentityMine had ventured into the field of Surface application development from its earlier days itself and still continues to deliver products with remarkable user experiences.



Continued to Microsoft Surface Application Development Environment.