Playing Smooth Streaming (.ism) videos in Silverlight…

I’m currently working on this Silverlight project, that will be a growing video archive that you might see go online in the close future. Unfortunately I can’t say more at the moment. But that is not what this post is about. The post is about playing Smooth Streaming videos in Silverlight.

I’m not that familiar with media streaming and video formats and so on, but when we started talking about playing video, I insisted on it being streaming. Http progressive download is just not that attractive and great… It will use up unnecessary bandwidth as well as cause certain “issues” at the client. Such as limiting the users ability to “scrub” though the video…

We quickly decided not to build our own player. Why do that, when people have already done it for you… So I pulled out Expression Encoder and got one of its premade players. I opened up the xap, pulled out the assemblies I needed and threw the rest away. That makes for a great base to inherit from. All I needed to do was basically redo the template and I was done.

However, besides the normal issues with the templating, we ran into some issues with the playback. We had the player in the application and initially did no streaming. Instead we fed it files from disk while developing the application. And then the day came when the customer had found a streaming supplier. It happened to be iStreamPlanet. And as they got involved it was time to get started with streaming. I thought we were going to use Windows Media Services and stream over the mms protocol. I had tried this before. But that wasn’t the case. iStreamPlanet streams using a new technique called Smooth Streaming and what they gave us were a couple of test urls. They had a file “.ism” file extension. Hmm…interesting…

I started looking around to see what this was and how it worked. And of course what implications it would have for us… Well… Luckily, the implications for us were not that big. The media player that Expression Blend uses, supports Smooth Streaming natively. However, we still had streaming issues, so I decided to look under the hood and see what was going on…what is Smooth Streaming? How does it work? Why is it cool? Why haven’t I built it myself?

Well, let’s start with what Smooth Streaming is. It is a technology that supports streaming from an IIS 7 server. That’s nothing new, but the way it works is kind of cool. At least this is the way that I have figured it works…

When you get a url to a .ism file, don’t try to browse to it straight up. That won’t work. What you need to do is get the manifest. This is available by appending “/manifest” at the end of the url. This manifest is basically an Xml file containing the different feeds available. There are several different feeds involved. Each feed has its own bitrate. So when the player is playing, it can actually switch between feeds with different bitrates while playing. So the quality of the streamed video actually fluctuates while playing. Actually, it is not the player that does the switching…but I will get to that later…

The player then starts downloading the bitrate it wants (once again…not the player…more to come). If the bitrate is too high for the available connection, the selected stream will change. So it will go to one with less or higher bitrate depending on bandwidth available.

How can it change in the middle of streaming? Well, it actually streams in a cool way. It downloads a little piece at the time. It isn’t an open connection constantly streaming. Instead the stream is more like a file on disk, that the player can request chunks from. So it can download a couple of kbs, called a chunk, and then play that while it requests the next chunk. So it is more like chunk reading a large file, than streaming. This technique makes it possible to figure out how fast we are actually downloading, as well as switch between streams by just downloading the next chunk from another stream…

And why is this cool? Well…it makes it possible to stream high quality video to those who have enough bandwidth, but still fallback to lower quality if needed. We are actually streaming 720p video at good quality. And that is streaming to New Zealand…which is not really the best connected place in the world.

This technology has been used by a lot of players already, including the Olympic Games and companies like Netflix. The Olympic Games used smooth streaming as it enabled them to quickly edit video on the fly using a Silverlight based application, and then deliver more content to the viewers faster and with the best available quality. And companies like Netflix use smooth streaming as they too need the ability to offer the best quality possible for each individual viewer looking to for the best quality when they watch TV shows online.

So…technically, what do we need to do? Do we need some new fancy control for Silverlight to play this? No…the MediaElement can actually play this. But not out of the box. The way that you get this going is by using a new type of source. This source can be extracted from the Encoder player, which is what I did. It is called AdaptiveStreamingSource and in the AdaptiveStreaming.dll assembly.

AdaptiveStreaming.dll is available inside the xap that Expression Encoder produces when asked to add a player to the encoded output. Just take that xap. Rename the .xap to .zip. Open and pull out the assemblies needed…

So what you need to do, is to reference the AdaptiveStreamingSource assembly. Then in code you create a new AdaptiveStreamingSource passing in the MediaElement and the Uri to the .ism file. However, you need to add the “/manifest” at the end of the Uri. Then you use this source to set the source of the MediaElement. After that, the MediaElement will play just as normal. The AdaptiveStreamingSource object also gives you access to a few new events that might be interesting. You can get notified of each chunk download, bitrate change, buffer start and so on…

public void Play(string url)
{
AdaptiveStreamingSource src = new AdaptiveStreamingSource(myMediaElement,new Uri(url + "/manifest"));
myMediaElement.SetSource(src);
myMediaElement.Play();
}

If you want to extend the Encoder player, I recommend you inherit from the ExpressionMediaPlayer.MediaPlayer class in the MediaPlayer.dll assembly. The MediaPlayer.dll assembly is available in the Encoder xap just like the AdaptiveStreaming.dll. If you chose to inherit from this class, you do not handle the setting of the MediaElement’s Source on your own. Instead you have to create a class implementing the IMediaStreamSourceFactory interface. This class is then used to set the base class’ MediaStreamSourceFactory property. In this class you are responsible for instantiating the AdaptiveStreamSource to use for the playing. The factory can look like this

private class AdaptiveStreamingSourceFactory : IMediaStreamSourceFactory
{
public virtual MediaStreamSource Create(MediaElement mediaElement, Uri uri)
{
return new AdaptiveStreamingSource(mediaElement, uri);
}
}

This factory can then be set on the base class in the constructor…

public Player()
{
this.SourceProperty = DependencyProperty.Register("Source", typeof(IPlayerSource), typeof(Player), new PropertyMetadata(null, new PropertyChangedCallback(Player.OnSourceChanged)));
base.MediaStreamSourceFactory = new AdaptiveStreamingSourceFactory();
}

As you see in the constructor I have a DependencyProperty called Source. When the source is changed, the following is code is executed

if (url.Contains(".ism"))
{
PlaylistItem item = new PlaylistItem();
item.MediaUrl = new Uri(url + "/manifest");
item.IsAdaptiveStreaming = true;
base.Playlist.Add(item);
}
else
{
PlaylistItem item = new PlaylistItem();
item.MediaUrl = new Uri(url);
item.IsAdaptiveStreaming = false;
base.Playlist.Add(item);
}

Setting the IsAdaptiveStreaming to true, causes the player to use the MediaStreamSourceFactory to get the source instead of just using the url.

I hope this helped…I haven’t been able to find that much information about this on the almighty Google… If you have any questions, you can just send a message or add a comment as usual…

Comments (15) -

Thanks very much! I was actually looking for this and wasn't able to find anything. Great post!

Hey Chris,

I'm assuming the site you've been working on is http://www.office2010themovie.com ?

It is looking very good!

I'm trying at the moment to get the same thing working with Silverlight 3 and the Encoder 3 templates as the underlying dll's and constructors have changed.

Checkout What's New In Encoder 3 - blogs.msdn.com/.../9828866.aspx

Hi Nigel!
Nice to see you in here! Yes, that is the one. I'm going to use some of the parts from that site in furture blog posts. Stay tuned... Smile

It's a good job.Thank you! But I can't experience it for I'm a rookie. Would you please send me the source code,or tell me how to add the codes in the source code detailedly.
By the way,the source code I used is included in "C:\Program Files\Microsoft Expression\Encoder 2\Templates\en\SL2Standard".

hi Chris could you please help me out by providing me  with the source code, I'm relatively new to silverlight as well. I'm using Smoothstreaming.dll, I cannot find AdaptiveStreaming.dll (expression encoder 3).
Thanks

Hi Kofi!
I've just had a quick look at the player in Expression Encoder 3, and it seems to be supporting adaptive streaming out of the box. It looks as if it uses a plug-in to support this if needed.
Looking a tone of the built in player templates, it has a SmoothStreaming.xap as well as the MediaPlayerTemplate.xap. So I guess you need to have a look at how this is supported. Unfortunately I don't have time at the moment...

Update...
The MediaPlayer object will automatically download the smooth streaming support if needed. So the SmoothStreaming.xap needs to be on the server as well. If the player is running in offline mode, it looks for a previously downloaded plug-in in the IsolatedStorage.
So theoretically, just inheriting from MediaPlayer, as mentioned in my post, should just work and no extra effort needed. Unless you manually want to manage this part and skip the deferred loading of the extra xap...

I am using expression encoder to create the video for me. When I use the default.html (generated by expression encoder) everything works fine. But when I change the MediaSource property of the MediaElement to be the exact same full URI it doesn't work.

For example, when the MediaSource="test.ism\Manifest" everything works. But when I change it to http://localhost/sl/test.ism\Manifest" I get a 4001 error.

This is a problem for me because our html files must be hosted on different servers then our Silverlight content. Is there something I am doing wrong?

Hi!
Well, it is hard to say what is wrong. My initial reaction is that it is correct and that you shouldn't have problems. But I have a question, are you getting to the html page through http://localhost as well, or is that open from the filesystem directly? I'm not too sure about the rules when it comes to media, but switching between file:// and http:// can cause crossdomain issues... Use Fiddler or Web Development Helper to see what requests are fired off. That should give you a clue...
Cheers!

Thanks for informative post. I was looking for similiar solution, and your article gave me idea, how it can be done with SilverLight.

Thanks!

I have few WMV files and few WMA files and have been playing them as a playlist via IIS Media Services.
Now I converted them to ISMV and ISMA files for variety of kbps connections to use Smooth Streaming.

Now, how can I play them in a playlist. I don't see any option there even for that?

Can I call ISM files one by one in c# by using PLAYLISTITEM where one ISM is connected with several ISMV or ISMA files.
Hence this way, I can individually play different content mixed audio and video encoded separately for smooth streaming via the playlist?

I mean would this work:

PlaylistItem item = new PlaylistItem();
    item.MediaUrl = new Uri("url/filename1.ism" + "/manifest");
    item.IsAdaptiveStreaming = true;
    base.Playlist.Add(item);

    item.MediaUrl = new Uri("url/filename2.ism" + "/manifest");
    item.IsAdaptiveStreaming = true;
    base.Playlist.Add(item);

    item.MediaUrl = new Uri("url/filename3.ism" + "/manifest");
    item.IsAdaptiveStreaming = true;
    base.Playlist.Add(item);

Here filename1.ism contains information of several ISMV files, filename1.ism contains information of several ISMA files...

Please confirm if this would work like a playlist rather than creating ONE SINGLE file as WMV by joining Video and Audio and then creating single ISM file?

Thanks.

Love,
Rishi

Hi Angelo!
To be honest, it has been quite some time since I wrote this and wince I worked with this code. But I think that you might experience some problems when splitting the video and audio into separate streams. I have however never tried it so I can't confirm if it works or not.
When it comes to playing Smooth Streaming though, I recommend looking at some newer things than what I use in this blog post. From a player perspective, I would recommend either looking at the player in Expression Encoder 3 or 4 or downloading the Smurf, Silverlight Media Framework, from Codeplex. It should have most of the stuff that you would need. It should be really quick to just create a simple proof of concept app for you scenario if you use the smurf...
Hope that helps in some way...
Cheers,
Chris

hi chris, do you know how to create the .ISMC file "by hand?" i ran encoder4 and while it created all the .ismv files, it didn't create the .ism or .ismc files. the transcoding took almost 3 hours!  

i was able to fake the .ism file from another output. that was simple enough, but i don't know what to do about the .ismc file. do you know? thanks, this was a good post.

Hi Gerardo!
I have actually never tried doing that. I have never managed to get Encoder to encode to smooth streaming without creating all the needed files...
Sorry...

There are some new resources that some of you might find useful:
* Smooth Streaming overview:  http://www.iis.net/SmoothStreaming
* Smooth Streaming demo:  http://www.iis.net/media/demo
* Smooth Streaming client (Silverlight Media Framework):  http://www.codeplex.com/smf
* Expression Encoder 4 (with built-in Smooth Streaming templates):  http://www.microsoft.com/expressionencoder

I hope you find these useful...

-Chris

Comments are closed