Silverlight 3 multi-touch development 101 – pt 1 - Pan

As I wrote in my previous blog post, I have wanted to try it out ever since I heard that it was available. But as I also mentioned, there has been a certain lack of access to multi-touch enabled devices. However, this is easily solved by using an iPod touch or iPhone, which is what I started out using. Unfortunately, that solution doesn’t really give you the right “feeling” as you are using it as a touchpad instead of actually “touching” the object.

Luckily, the nice Chris Auld went to PDC and got a multi-touch enabled laptop that he happily lent to me (unfortunately I believe he expects me to give it back at some point). And with this new toy, I got started… (If you are wondering what hardware I am actually using it, it is this. And I highly recommend it…)

The first thing i did was to open up a new project and have a look at what Microsoft gave us and how we can use it. I did decide to go with the “raw” API instead of adding a helpful library in between. Why? Well…I really wanted to know what is going on under the hood, before I let someone tweak it…but that’s just me.

Unfortunately, Microsoft didn’t really give us that much. To be honest, all they gave us was a way to get the position of the different input points. From that, we have to build our own solution. That means that if you want gesture support or anything like that, you got your work cut out for you. But as I said, there are libraries available to help out if you need it. And to be honest, unless you like math and know your trigonometry, you won’t get very far. Having that said, I am not the sharpest math person in the world, and still got my sample to work. All though, it took four times as long as I had expected…

So…to cut through all of my talk, what we got from Microsoft was a single static object called Touch. Well…that’s ok…a single object can still give us a lot of functionality…well…it can…but Touch doesn’t. It gives us one event called FrameReported. So, from that I concluded that I needed a wrapper that added the functionality I wanted, which was basically the standard touch features – Pan, Rotate and Zoom. And the application to build is the usual one, the image viewer that makes it possible to manipulate an image on screen using your fingers. We have all seen it before, but the main thing right now is the TouchManager. That fact that I use it for something as mundane as an image viewer, doesn’t mean that it can’t be very useful for other applications.

This first part will cover Pan. While the next posts will cover the Rotate and Zoom functionality.

I start off my TouchManager by creating a simple class. The constructor takes a single argument, the element that is to be manipulated. The reason that my manager needs a reference to the element being manipulated is that some of my calculations need to be based on local coordinates inside that element and some based on global coordinates. The TouchManager stores the element reference and attaches an eventhandler to the FrameReported event. The class also contains 2 global variables of type List<PointDefinition>. I will cover PointDefinition later, but for now, all you need to know is that the 2 lists contain a set of objects that represents the touch points reported by the FrameReported event. One list that contains them all (sort of), and one that contains the points that are actually used for the “gestures”.

The TouchManager will only use 2 points for the “gestures”, but the input device could potentially support more points. I think the MacBook supports 11 points, and I know that the iPod Touch supports at least 5. The Acer laptop I am using only supports 2 though…I think…anyhow, I need to be prepared for the possibility that the input device sends more points than I expect or want…

public class TouchManager
{
UIElement _localElement;
List<PointDefinition> _points = new List<PointDefinition>();
List<PointDefinition> _currentlyActivePoints = new List<PointDefinition>();

public TouchManager(UIElement localElement)
{
_localElement = localElement;
Touch.FrameReported += new TouchFrameEventHandler(OnFrameReported);
}
}

 

Before I can go on and show the OnFrameReported method, I need to show a supporting class that I use. It is the PointDefinition class. It is a very simple little class. It has a static method and 5 properties. The 5 properties consists of one integer, the ID of the touch point that is represents, and 4 Points. The local point before the latest movement, and the local point after. And the same for the global points. The static method is a factory method that helps us when creating a new object, by setting the ID and the new points.

protected class PointDefinition
{
Point _newGlobalPosition;
Point _newLocalPosition;

public static PointDefinition Create(TouchPoint localP, TouchPoint globalP)
{
PointDefinition pd = new PointDefinition();
pd.ID = localP.TouchDevice.Id;
pd.NewLocalPosition = localP.Position;
pd.NewGlobalPosition = globalP.Position;
return pd;
}
public int ID { get; set; }
public Point LastGlobalPosition { get; private set; }
public Point NewGlobalPosition
{
get { return _newGlobalPosition; }
set {
LastGlobalPosition = NewGlobalPosition;
_newGlobalPosition = value;
}
}
public Point LastLocalPosition { get; private set; }
public Point NewLocalPosition
{
get { return _newLocalPosition; }
set
{
LastLocalPosition = NewLocalPosition;
_newLocalPosition = value;
}
}
}

Oh yeah…it is also only being used internally and never visible to the consumer of the TouchManager, so I declared it as protected inside the TouchManager class. That way, it never “leaks” out, and doesn’t clutter my intellisense…

After that little thing is out of the way, it is time to start looking at the OnFrameReported method. It starts off by getting the TouchPoints for both the local and global coordinates. For global coordinates, I get the points relative to the application’s RootVisual.

protected virtual void OnFrameReported(object sender, TouchFrameEventArgs e)
{
TouchPointCollection globalPoints = e.GetTouchPoints(Application.Current.RootVisual);
TouchPointCollection localPoints = e.GetTouchPoints(_localElement);
...
}

 

It then loops through one point at the time, checking the points Action. Each point will indicate if it has been added (TouchAction.Down), moved (TouchAction.Moved) or removed (TouchAction.Up). I happen to loop over the globalPoints collection, but it doesn’t really matter which one I choose. I will still be handling all of the changes based on ID.

The loop starts by requesting the corresponding PointDefinition object based on the TouchPoint’s ID. The TouchPoint’s ID is available through TouchPoint.TouchDevice.Id. The ID as such is completely irrelevant. The only thing we use it for is as reference when the point has been changed.

If the ID is not available in the _points' list, it is either new or not interesting. If its Action isn’t TouchAction.Down, then it isn’t new and thus not interesting.

foreach (var p in globalPoints)
{
PointDefinition point = GetPointDefinition(p.TouchDevice.Id);
if (point == null && p.Action != TouchAction.Down)
return;
...
}

 

The little helper method called GetPointDefinition is just wrapping a Linq statement, making the code more readable…

protected PointDefinition GetPointDefinition(int id)
{
return _points.FirstOrDefault(p => p.ID == id);
}

 

So…what do I really mean with a point not being interested? Well, I only care about points that are actually inside the managed element. An element should not be affected by points that are outside of its bounds…thus any point outside it is uninteresting. This is checked in the next part of the loop, which is a switch statement. It “switches” on the TouchPoint’s Action.

If the Action is Down, I verify that the point is actually “on” the element. If it is, I create a new PointDefinition for that point and add it to the _points collection. Since I am looping over the global points, I use a Linq statement to get hold of the corresponding local point.

If the Action is Up, I remove it from the _points collection, as well as the _currentlyActivePoints collection if needed.

And finally, if the action is Moved I update the PointDefinition’s New<Local/Global>Position.

switch (p.Action)
{
case TouchAction.Down:
if (!IsInsideElement(p))
continue;
_points.Add(PointDefinition.Create(localPoints.FirstOrDefault(z=>z.TouchDevice.Id == p.TouchDevice.Id), p));
break;
case TouchAction.Up:
_points.Remove(point);
if (_currentlyActivePoints.Contains(point))
_currentlyActivePoints.Remove(point);
break;
default:
point.NewGlobalPosition = p.Position;
point.NewLocalPosition = localPoints.FirstOrDefault(z => z.TouchDevice.Id == p.TouchDevice.Id).Position;
break;
}

 

The final step in the handler is to actually handle any movement. But before I can do this, I check my _currentlyActivePoints collection. If it has 2 points, it is time to handle the change, but if not, I try to get 2 points from the _points collection, since I might have added it during the loop. But, if it has been added during the loop, it means that at least one of the points were just added and that no “gesture” has been performed, so I exit the handler.

if (_currentlyActivePoints.Count == 2)
HandleMovement(_currentlyActivePoints);
else
_currentlyActivePoints = new List<PointDefinition>(_points.Take(2));

 

And now the fun starts. Now the trigonometry from school should kick in and help you loads. Unfortunately, it was a while since I did trigonometry, and I generally don’t like “straight up” math that much. So the math part of this might be possible to improve a lot. If you feel that is the case, please drop me a line and help out a fellow developer… :)

The first step is to make sure that all of the points have actually been moved. They should definitely have been moved before entering this method, but better safe than sorry.

protected virtual void HandleMovement(List<PointDefinition> points)
{
foreach (var p in points)
if (p.LastGlobalPosition == default(Point))
return;
...
}

After that little check, I go on to find the center point between the 2 “finger points”. I want this point for both the "”before” and “after” points. To do this, I created a helper class. It isn’t actually created to help out with that particular calculation, but it wraps all of my calculations. It is a static class called TrigUtility. You will see more of it later, but I have decided to introduce the functionality as needed…

 

The TrigUtility method used looks like this

public static class TrigUtility
{
...
public static Point GetCenterOfPoints(Point point1, Point point2)
{
double centerX = (point1.X + point2.X) / 2;
double centerY = (point1.Y + point2.Y) / 2;
return new Point(centerX, centerY);
}
...
}

 

From the before and after points, I can easily figure out the Pan. As soon as I got that information, I expose it through an event called Pan (how clever…).

Point lastCenterPoint = TrigUtility.GetCenterOfPoints(points[0].LastGlobalPosition, points[1].LastGlobalPosition);
Point newCenterPoint = TrigUtility.GetCenterOfPoints(points[0].NewGlobalPosition, points[1].NewGlobalPosition);
Point movement = new Point(newCenterPoint.X - lastCenterPoint.X, newCenterPoint.Y - lastCenterPoint.Y);
OnPan(movement);
protected virtual void OnPan(Point movement)
{
if (Pan != null)
Pan(this, new TouchEventArgs<Point>(movement, GetActivePointsLocal(), GetActivePointsGlobal()));
}
public event EventHandler<TouchEventArgs<Point>> Pan;

As you can see, the Pan event, as well as the Rotate and Zoom event that will be introduced later, passes its information to the consumer using a custom EventArgs class called TouchEventArgs<T>. This custom EventArgs class exposes the change (pan, rotation or zoom) through a generic Payload property. It also exposes active points as local as well global coordinates.

public class TouchEventArgs<T> : EventArgs
{
public TouchEventArgs(T payload, Point[] localPoints, Point[] globalPoints)
{
Payload = payload;
LocalPoints = localPoints;
GlobalPoints = globalPoints;
}
public T Payload
{
get;
private set;
}
public Point[] LocalPoints
{
get;
private set;
}
public Point[] GlobalPoints
{
get;
private set;
}
}

 

After having tackled the TouchManager’s part of the Pan, it is time to take a look at the client. The client in this case is a fairly simple little thing. The Xaml looks like this

<UserControl x:Class="DarksideCookie.iPad.Modules.TouchDemo.Views.TouchDemoView"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<Grid x:Name="LayoutRoot">
<Grid x:Name="TranslatedContainer">
<Grid.RenderTransform>
<TransformGroup>
<RotateTransform Angle="0" x:Name="Rotation" />
<TranslateTransform x:Name="Translation" />
<ScaleTransform x:Name="Scaling" />
</TransformGroup>
</Grid.RenderTransform>
<Border BorderBrush="Black" BorderThickness="1" HorizontalAlignment="Center" VerticalAlignment="Center">
<Image Source="../Images/VistaImage.jpg" Stretch="None" />
</Border>
</Grid>
</Grid>
</UserControl>

As you can see, it is just a Grid (inside the root grid) called TranslatedContainer, which might not be the best name in the world, but it will have to suffice for this demo. It contains an image and has a couple of Transform objects defined. These will be manipulates based on the event information from the TouchManager.

The code behind is not a lot more complicated at the moment… In the constructor, a new TouchManager is created and placed in a global member and then the eventhandler for the Pan event is attached up.

public partial class TouchDemoView : UserControl
{
TouchManager _tm;

public TouchDemoView()
{
InitializeComponent();
_tm = new TouchManager(TranslatedContainer);
_tm.Pan += OnPan;
}

void OnPan(object sender, TouchEventArgs<Point> e)
{
Translation.X += e.Payload.X / Scaling.ScaleX;
Translation.Y += e.Payload.Y / Scaling.ScaleY;
}
}

 

The OnPan event handler translates the container by the passed in X and Y values. It also takes into consideration any scaling of the container. It might look odd that I divide by the Scaling instead of multiplying, but the reason is quite simple. The Pan calculations have all been done based on global coordinates instead of local. The reason for this is that using local coordinates when translating it causes it to jump back and forth. The TouchManager gets information that the touch points have moved, the client moves the container based on this, and all of the sudden the local coordinates are changed due to the translation, so the manager thinks the touch points have moved again and tells the client to move it back… So, we have to use global coordinates. And because of this, the movement will have to be divided by the scale instead of multiplied… I hope that makes sense…

Note that the translation is done using +=, that means that it adds to the existing translation. All our changes to the container will be relative to its current situation. Cause the TouchManager is continuously feeding us small changes and not the complete changes to the object as such…if that makes sense…

That’s it for now. If you implement this, you should be able to move your element around the screen using 2 fingers on your device…hmm…that came out as it should, but still sounds kind of wrong…but you know what I mean…

Next time, I will tackle something a little harder, the rotation…

Until then, have a great time and don’t hesitate to contact me is you have any questions!

Comments are closed