I have recently posted a few posts on how to use some of the new features of the Azure Service Bus. They seem to have been somewhat popular, which is fun. They are however very light weight introductions, and not that I am going to dig a whole lot deeper at the moment, but there are a few little things I want to mention.Mainly around brokered messages.
As you know from the previous posts, a brokered message, is a message that is sent to the bus from a client, and picked up by a service at some point. The message can contain a body, which could be more or less any class that you would like, as well as metadata about the message. The only thing to remember with those things, is the fact that the message size is limited to 256kb.
You might also want to keep in mind that all messages sent to a Queue, or a Topic for that matter, is stored until someone picks it up, or it times out. Obviously, Azure can’t take an infinite load, so these entities are equipped with a max size as well, which can be defined at time of creation. The limits are between 1 and 5 Gb. If a Queue or Topic exceeds this size, the incoming messages will be rejected and an Exception will be thrown by the client.
You are also limited to a maximum of 2000 subscriptions per Topic and a maximum of 10.000 topics/queues per namespace. However, that is a LOT of topics and queues… If you need more, I guess you will just have to span across multiple namespaces.
If you have a system where you expect a ridiculously high load, or a very slow or disconnected service, and you can afford to loose messages, then you can make sure that a message only stays in a store for a certain amount of time. By setting the TimeToLive property on a BrokeredMessage, you tell the system that if the message isn’t delivered within a certain timespan, it should be discarded, or dead lettered (I will get back to this).
You can also set the DefaultMessageTimeToLive property on a Queue, Topic or Subscription. This will set a default TimeSpan to use, which can be overridden by a TTL property set on an individual message. However, a message can never define a longer TTL than the destination, that is it can never be longer than the default value.
Another important concept is called “dead lettering”. Dead lettering a message basically means taking a message out of the current store and putting it in a specific sub-queue that is used to store messages that cannot be handle for some reason.
A message could be dead lettered if the previously mentioned TTL expires. This is the case if the the target, for example a Queue, has the property EnableDeadLetteringOnMessageExpiration set to True. This causes expired messages to go in the dead letter queue instead of being deleted.
The sub-queue for dead lettered messages is named $DeadLetterQueue. So if the Queue name is MyQueue, dead lettered messages are placed in a sub-queue called MyQueue/$DeadLetterQueue.
There are however other reasons that a message might get dead lettered. When retrieving messages from a store, the client can do this in 2 ways, “PeekLock” and “RecieveAndDelete”, with “ReceiveAndDelete” being the default. When using ReceiveAndDelete, the message is pulled from the store and deleted, very simple, but if something in the processing fails, the message will be lost.
If you instead use PeekLock, the message is read from the store for handling, but not deleted. Instead, it goes into hiding for a set amount of time (defined by LockDuration, 120 secs by default I think). If the service does not tell the system that handling went fine, the message shows up again after the defined time.
So the service is responsible for calling one of several methods on the message before ending the processing, and before the LockDuration has elapsed. The choices that the service have are Complete(), Abandon() or Deadletter(). Not calling any of the methods, is the same as Abandon(), but not until the LockDuration time has elapsed.
Complete() will tell the store that processing went fine, which in turn will delete the message from the store.
Abandon() will tell the store that the service could not process the message, and it will pop back up in the queue straight away for new processing. The reason for calling this could be many, but it is basically a way to put the message back on the queue for another round of processing.
Deadletter() will send the message to the dead letter queue straight away. This is if something is really wrong with the message.
However, the fact that not calling any method being the same as calling Abandon() and returning the message to the store is a bit dangerous. If a service receives the message, but the message contains an error, or the service is flawed, it might fail due to an Exception being thrown. This would from the Service Bus point of view just mean that Complete() wasn’t called, and the message would reappear in a little while, potentially causing another error.
A message that keeps popping up and causing errors is called a “poison message”, as it poisons the system with errors every minute or two. As a response to this, the Service Bus keeps track of how many times a message has been delivered. If the delivery count exceeds the store’s MaxDeliveryCount, the message is considered a “poison message” and is sent to the dead letter queue.
The delivery count is available on the BrokeredMessage as the property with the not so surprising name of DeliveryCount.
It is however VERY important that you clean up the dead letter queue. Any message in this queue will count towards the max size of the store, and if it goes beyond the defined size limit, the store will refuse incoming messages.
Receiving and cleaning out the dead letter queue is done in the same way as you would clean out any other queue.
You can use the QueueClient/SubscriptionClient FormatDeadLetterPath() method to get the path to the dead letter queue.
string path = QueueClient.FormatDeadLetterPath("MyQueue");
QueueClient client = factory.CreateQueueClient(path);
while (true)
{
BrokeredMessage msg = client.Receive();
if (msg != null)
{
Console.WriteLine("Dead lettered message");
msg.Complete();
}
}
I hope this extra information is useful. Knowing some of this will hopefully make some things easier to understand. It will also make you look less like a tool when people start conversations about dead lettering and things.
If you have any questions, don’t hesitate to make a comment or drop me a line. There is no code for this post though… 