A couple of weeks ago, Microsoft asked me to add closed captioning to the videos on the www.office2010themovie.com. And no…not automated subtitles that converts to what is said into text, like YouTube does…just subtitles from a file. This is not very complicated, not even when you add the fact that the videos are actually being streamed using Smooth Streaming. But I wanted roll it al into a control, to make it fast and easy to add subtitles to any MediaElement in future projects…
The first thing I needed to solve was the parsing of the subtitle files. There are a lot of different subtitle formats available. Luckily Microsoft, for different reasons, wanted to use a format called SAMI. So my first action was to figure out what the SAMI format looked like, and how to parse it. So I Googled it and found out that it is basically a markup based format that resembles XML. It can be produced as valid XML, but the spec is a lot looser. It does for example not demand end elements. So it is sort of like old school HTML, which means that it is totally impossible to parse in a simple way. Luckily, I could get Microsoft to supply me with SAMI files that are XML compliant. They look like this:
<SAMI>
<HEAD>
<Title>SAMI</Title>
<STYLE TYPE="text/css">
<!--
P {font-size: 12pt;
font-family: Arial, sans-serif; font-style: normal; font-weight: normal;
color: #FFFFFF;}
.ENCC {Name: English; lang: en; SAMI_Type: CC;}
-->
</STYLE>
</HEAD>
<BODY>
<SYNC Start= "0">
<P Class="ENCC">Subtitle 1</P>
</SYNC>
<SYNC Start= "5000">
<P Class="ENCC">Subtitle 2</P>
</SYNC>
<SYNC Start= "9000">
<P Class="ENCC">Subtitle 3</P>
</SYNC>
</BODY>
</SAMI>
As you can see, it looks a lot like HTML. It even uses CSS for styling (which I blatantly ignore on this case). Each SYNC element has a Start attribute that corresponds to when the subtitle is supposed to be shown. The number is the time span from the start of the video expressed as milliseconds. If you want to just hide a subtitle without showing a new one, you add a SYNC with a Start attribute set to the when it is to be hidden and a content consisting of a P tag with an inside. Basically an empty subtitle causing the previous subtitle to be hidden…
So…plan of attack…? Well, my idea was to build a custom control that would be placed on top of the MediaElement. As long as the control is transparent except for the lines of text that are to be shown, it will look as if there is nothing but the subtitles. The only thing to remember is that videos are often shown in 2 different ways, as a media player in a bigger application and as a full screen view. So I decided to add 2 visual states, Normal and Fullscreen. to the Subtitle control. So the template that I decided to use looks like this
<Style TargetType="local:Subtitle">
<Setter Property="Margin" Value="10" />
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="local:Subtitle">
<Grid>
<vsm:VisualStateManager.VisualStateGroups>
<vsm:VisualStateGroup x:Name="FullscreenModes">
<vsm:VisualState x:Name="Normal">
</vsm:VisualState>
<vsm:VisualState x:Name="Fullscreen">
<Storyboard>
<ObjectAnimationUsingKeyFrames Storyboard.TargetName="itemsControl" Storyboard.TargetProperty="Visibility">
<DiscreteObjectKeyFrame KeyTime="0">
<DiscreteObjectKeyFrame.Value>
<Visibility>Collapsed</Visibility>
</DiscreteObjectKeyFrame.Value>
</DiscreteObjectKeyFrame>
</ObjectAnimationUsingKeyFrames>
<ObjectAnimationUsingKeyFrames Storyboard.TargetName="itemsControl1" Storyboard.TargetProperty="Visibility">
<DiscreteObjectKeyFrame KeyTime="0">
<DiscreteObjectKeyFrame.Value>
<Visibility>Visible</Visibility>
</DiscreteObjectKeyFrame.Value>
</DiscreteObjectKeyFrame>
</ObjectAnimationUsingKeyFrames>
</Storyboard>
</vsm:VisualState>
</vsm:VisualStateGroup>
</vsm:VisualStateManager.VisualStateGroups>
<ItemsControl x:Name="itemsControl" HorizontalAlignment="Center" VerticalAlignment="Bottom"
ItemsSource="{TemplateBinding SubtitleLines}" Margin="0,0,0,10">
<ItemsControl.ItemTemplate>
<DataTemplate>
<Border HorizontalAlignment="Left" Margin="0,2"
Background="#99000000">
<TextBlock Text="{Binding}" FontWeight="Bold" Margin="6,0"
Foreground="White" FontSize="16" />
</Border>
</DataTemplate>
</ItemsControl.ItemTemplate>
</ItemsControl>
<ItemsControl x:Name="itemsControl1" HorizontalAlignment="Center" VerticalAlignment="Bottom"
ItemsSource="{TemplateBinding SubtitleLines}" Visibility="Collapsed" Margin="0,0,0,30">
<ItemsControl.ItemTemplate>
<DataTemplate>
<Border HorizontalAlignment="Left" Margin="0,2"
Background="#99000000">
<TextBlock Text="{Binding}" FontWeight="Bold" Margin="6,0"
Foreground="White" FontSize="35" />
</Border>
</DataTemplate>
</ItemsControl.ItemTemplate>
</ItemsControl>
</Grid>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
As you can see, there is nothing complicated at all. It basically consists of 2 ItemsControls that are shown one at the time depending on the state. The ItemTemplates are also simple. They are nothing more complicated than a Border with a semi-transparent background and a TextBlock inside. This will give a nice background to the text, making it readable on any background.
The two different ItemsControls have very few of differences. The fullscreen one has a bigger font. My original plan was to have a single ItemsControl and modify the font size using template binding. Unfortunately, that didn’t work too well. It is hard to get contents inside a DataTemplate template bound to the control, so I decided to use a hidden control and an element-to-element bindings. That worked fine in Silverlight 4, but not in Silverlight 3. So I am not going to show that solution, and instead just demo the state driven version that actually works…
The code behind the control is not hugely complicated, but it exposes 5 dependency properties, so let’s have a look at those…
The first property has actually already been hinted in the ControlTemplate. It is called SubtitleLines and is of type string[]. It is template bound to the ItemsControls used in the ControlTemplate. So whenever it is set, the currently visible ItemsControl will display those lines of text. This property is never set externally and will instead always be set from the code in the control.
public static readonly DependencyProperty SubtitleLinesProperty =
DependencyProperty.Register("SubtitleLines", typeof(string[]), typeof(Subtitle), new PropertyMetadata(null));
private string[] SubtitleLines
{
get { return (string[])GetValue(SubtitleLinesProperty); }
set { SetValue(SubtitleLinesProperty, value); }
}
The next property up is called SubtitleUri. It is as you might have guessed a property that will be set to the Uri of the subtitle file (in my case a SAMI file). When the property is changed, it does a few things. It starts by setting the current SubtitleLines to null, removing any subtitles on screen. Next, it checks to see if it has a value. If not, is clears out a local variable that holds the current subtitles, making sure that no subtitles are shown. If it has a value, it fires up a WebClient and downloads the file. It then stores the result in a global variable before checking to see if a subtitle parser has been defined. If a parser is defined, it is used to parse the subtitles that have been downloaded. I will cover the parser later. Right now, all you need to know is that it is an object that takes a string and returns subtitle objects…
public static readonly DependencyProperty SubtitleUriProperty =
DependencyProperty.Register("SubtitleUri", typeof(Uri), typeof(Subtitle), new PropertyMetadata(null,
new PropertyChangedCallback(OnSubtitleUriPropertyChanged)));
List<ISubtitleItem> _subtitleItems;
string _subtitles;
private static void OnSubtitleUriPropertyChanged(DependencyObject sender, DependencyPropertyChangedEventArgs e)
{
((Subtitle)sender).OnSubtitleUriPropertyChanged(e);
}
private void OnSubtitleUriPropertyChanged(DependencyPropertyChangedEventArgs e)
{
SubtitleLines = null;
if (e.NewValue != null)
{
_subtitleItems = new List<ISubtitleItem>();
WebClient client = new WebClient();
client.DownloadStringCompleted += (s, args) =>
{
if (args.Cancelled || args.Error != null)
return;
_subtitles = args.Result;
if (Parser != null)
{
Parser.Parse(_subtitles, (subtitle) => _subtitleItems.Add(subtitle));
}
};
client.DownloadStringAsync((Uri)e.NewValue);
}
else
{
_subtitleItems = null;
}
}
public Uri SubtitleUri
{
get { return (Uri)GetValue(SubtitleUriProperty); }
set { SetValue(SubtitleUriProperty, value); }
}
The parser that is being used is an instance of an class implementing and interface called ISubtitleParser. The ISubtitleParser interface declares a single method called Parse(), which takes two parameters. The parameters consists of a string containing the downloaded subtitles and an Action<ISubtitleItem> callback that is called once for each parsed subtitle.
public interface ISubtitleParser
{
void Parse(string subtitles, Action<ISubtitleItem> callback);
}
The ISubtitleItem interface is almost as simple. It declares 3 methods, an int called Show, an int called Hide and a string[] called TextLines. The Show and Hide properties represents the milliseconds into the movie that the subtitle should be shown and hidden and the TextLines array contains the lines of text that make up the subtitle.
public interface ISubtitleItem
{
int Show { get; }
int Hide { get; }
string[] TextLines { get; }
}
So…why have I defined these parts as interfaces instead of just an implementation? Well, it makes it possible to plug in any subtitle parser that you whish to use, making it possible to use what ever subtitle format you wish. In this post I will only show an implementation for SAMI files, but it is still possible…
The next dependency property to look at on the Subtitle control is of type ISubtitleParser. In its change callback, it checks the new value to see if it has been set to null and if the _subtitles string has been set (a subtitle file has been downloaded). Otherwise, it just sets the _subtitleItems to null and exits. If there on the other hand is a parser and there is a string with subtitles, it initializes the _subtitleItems variable and then tells the parser to parse the subtitles…
public static readonly DependencyProperty ParserProperty =
DependencyProperty.Register("Parser", typeof(ISubtitleParser), typeof(Subtitle), new PropertyMetadata(null,
new PropertyChangedCallback(OnParserPropertyChanged)));
private static void OnParserPropertyChanged(DependencyObject sender, DependencyPropertyChangedEventArgs e)
{
((Subtitle)sender).OnParserPropertyChanged(e);
}
private void OnParserPropertyChanged(DependencyPropertyChangedEventArgs e)
{
if (e.NewValue != null && _subtitles != null)
{
_subtitleItems = new List<ISubtitleItem>();
((ISubtitleParser)e.NewValue).Parse(_subtitles, (subtitle) => _subtitleItems.Add(subtitle));
}
else if (e.NewValue == null)
{
_subtitleItems = null;
}
}
public ISubtitleParser Parser
{
get { return (ISubtitleParser)GetValue(ParserProperty); }
set { SetValue(ParserProperty, value); }
}
So…now we have a way to download a subtitle file, parse it and expose it to the UI. So how do we actually get something to happen in the UI? Well, we need at least one more property. I have however decided to add 2. One of them is just sort of an overload though…
The first property is of type Nullable<TimeSpan> and is called MediaPosition, while the second one, of type MediaElement, is called (not too surprisingly) MediaElement. These overlap in the way that you can either data bind the MediaPosition and use this to set the current position in the movie or you can element to element bind a MediaElement and have the Subtitle control automatically read the media position from it. The MediaElement property code looks like this
public static readonly DependencyProperty MediaElementProperty =
DependencyProperty.Register("MediaElement", typeof(MediaElement), typeof(Subtitle), new PropertyMetadata(null,
new PropertyChangedCallback(OnMediaElementPropertyChanged)));
private static void OnMediaElementPropertyChanged(DependencyObject sender, DependencyPropertyChangedEventArgs e)
{
((Subtitle)sender).OnMediaElementPropertyChanged(e);
}
private void OnMediaElementPropertyChanged(DependencyPropertyChangedEventArgs e)
{
if (e.NewValue != null)
{
SetBinding(MediaPositionProperty, new System.Windows.Data.Binding("Position") { Source = e.NewValue, Mode = System.Windows.Data.BindingMode.TwoWay });
}
else
{
MediaPosition = null;
}
}
public MediaElement MediaElement
{
get { return (MediaElement)GetValue(MediaElementProperty); }
set { SetValue(MediaElementProperty, value); }
}
As you can see in the code, the MediaElement property changed code just fallbacks on the MediaPosition property by wiring up a data binding to the MediaPosition property. It doesn’t make a whole lot of sense as you could just bind to the MediaPosition property directly, but it looks pretty sweet in the Xaml when you just add a Subtitle control and hook it up to the MediaElement.
All the actual stuff happens in the MediaPosition property, or rather in the change callback method. In there, it starts by checking if the current control is actually visible. If not, it just returns. No need to waste processing in that case. Especially since the MediaPosition will be updated VERY often. And I mean VERY often. So it is worth doing some extra checks before going ahead with any functionality.
Next, it checks if there are any subtitles loaded at all. If not, then just return and ignore it all… If there are subtitles it figures out how many milliseconds into the movie it is. It then checks to see if there is a subtitle showing currently. If there is, it checks to see if it needs changing. If not, it once again just returns.
However, if it does need changing, the code goes ahead and makes a Linq query over the List<T> of subtitles to find the one to show at the moment. It then sets another DependencyProperty called SubtitleLines, which is used as ItemsSource for the ItemsControls in the template using TemplateBindings.
public static readonly DependencyProperty MediaPositionProperty =
DependencyProperty.Register("MediaPosition", typeof(TimeSpan?), typeof(Subtitle), new PropertyMetadata(null,
new PropertyChangedCallback(OnMediaPositionPropertyChanged)));
private static void OnMediaPositionPropertyChanged(DependencyObject sender, DependencyPropertyChangedEventArgs e)
{
((Subtitle)sender).OnMediaPositionPropertyChanged(e);
}
private void OnMediaPositionPropertyChanged(DependencyPropertyChangedEventArgs e)
{
if (Visibility == Visibility.Collapsed)
return;
if (_subtitleItems == null)
return;
int value = (int)((TimeSpan)e.NewValue).TotalMilliseconds;
if (_currentSubtitle != null)
{
if (_currentSubtitle.Show <= value && _currentSubtitle.Hide >= value)
return;
}
_currentSubtitle = (from i in _subtitleItems where i.Show <= value && i.Hide >= value select i).FirstOrDefault();
if (_currentSubtitle != null)
SubtitleLines = _currentSubtitle.TextLines;
else
SubtitleLines = null;
}
public TimeSpan? MediaPosition
{
get { return (TimeSpan)GetValue(MediaPositionProperty); }
set { SetValue(MediaPositionProperty, value); }
}
So…that is actually the whole control. The last thing to have a look at is the SAMI implementation of the ISubtitleParser and ISubtitleItem. And guess what…I called it SAMISubTitleParser. It has no other methods than the Parse dictated by the interface. In this method, I parse the passed in string using an XmlReader. I decided to use an XmlReader instead of XDocument, since using that class includes adding a reference to System.Xml.Linq, which adds a substantial amount of download to the Xap package. And yes…I know that I am paranoid about download size, but it is just because I started web development when download size was really important. So, go ahead and use XDocument if you want to. It makes it easier, but on the other hand, parsing this format is quite easy using the XmlReader…
public class SAMISubTitleParser : ISubtitleParser
{
public void Parse(string subtitles, Action<ISubtitleItem> callback)
{
string subtitle = subtitles.Replace(" ", "");
XmlReader reader = XmlReader.Create(new StringReader(subtitle));
reader.ReadToDescendant("SYNC");
do
{
SubtitleItem item = SubtitleItem.Create(reader);
if (item != null)
callback(item);
} while (reader.ReadState != ReadState.EndOfFile);
reader.Close();
}
}
As you can see in the snippet above, I have another class called SubtitleItem. It is a class that implements ISubtitleItem. It also has a factory method that helps with the parsing…
public class SubtitleItem : ISubtitleItem
{
private SubtitleItem(int show, int hide, string[] textLines)
{
Show = show;
Hide = hide;
TextLines = textLines;
}
public static SubtitleItem Create(XmlReader reader)
{
reader.MoveToAttribute("Start");
int show = int.Parse(reader.Value);
reader.MoveToElement();
reader.ReadToDescendant("P");
List<string> subs = new List<string>();
foreach (var sub in reader.ReadInnerXml().Split(new string[] { "<br />" }, StringSplitOptions.RemoveEmptyEntries))
{
string s = sub.Trim();
while (s.Length > 45)
{
int index = s.LastIndexOfAny(new char[] { ' ', '.', '-' }, 45);
if (index > 0)
{
subs.Add(s.Substring(0, index));
s = s.Substring(index + 1);
}
}
subs.Add(s);
}
reader.ReadToFollowing("SYNC");
if (subs.Count == 0)
return null;
string[] textLines = subs.ToArray();
reader.MoveToAttribute("Start");
int hide = int.Parse(reader.Value);
return new SubtitleItem(show, hide, textLines);
}
public int Show { get; private set; }
public int Hide { get; private set; }
public string[] TextLines { get; private set; }
}
That’s it! All that is left now is to use it…which is REALLY simple. The Xaml looks like this
<ctrl:Subtitle x:Name="subtitleControl" SubtitleUri="{Binding SubtitleUri}">
<ctrl:Subtitle.Parser>
<ctrlUtils:SAMISubTitleParser />
</ctrl:Subtitle.Parser>
</ctrl:Subtitle>
Sweet right? A reusable control for displaying subtitles with your videos…
Code for download: Subtitle.zip (3.95 kb)