Architecting Distributed Apps for Azure – Part 3

This blog covers the following topics related to messaging communication:

  • The benefits of messaging communication
  • Fault-tolerant message processing

The benefits of messaging communication

Messaging is a key strategy employed in many distributed environments such as the cloud. It enables applications and services to communicate and cooperate, and can help to build scalable and resilient solutions. Messaging supports asynchronous operations, enabling you to decouple a process that consumes a service from the process that implements the service.

Benefits of messaging communication:

  • Resource efficient
    • The client does not wait for the service reply, thus no blocked threads or long lived locks.
    • Service instances pull work, vs. clients constantly pushing work to instances that may be busy serving other requests. This model has been proven mathematically to be the most efficient servicing model, also known as single queue, multiple servers.
      • The queue service acts as an intermediary, which may help with service discovery issues.
  • Resilient: The client and service instances can be taken down or move to another server without adverse effect since the messages will be kept by the messaging service.
    • If a service instance fails, another instance processes the message. This means, the single message message can be delivered multiple times and not in order. This implies the services is idempotent.
    • The clients and the services can be taken offline without message loss. Once the service or the client is deployed it will start pulling in the message on the messaging service
  • Elastic: The queue length can be used to determine the need to scale the service up and down. If the queue length is getting higher over time, the orchestrator can decide to increase the instance number, and if it is getting smaller over time, the orchestrator can decrease the number of instances.

Fault-tolerant message processing

In order to have messaging communication, the message processing needs to be fault tolerant.

Processing messages in this way requires that the queuing service support some features, plus the queue client must use them to manage the messages.

First the service needs to support dequeue counts on messages, and make the messages reappear on the queue if they are not explicitly deleted within a certain period of time after being dequeued (invisibility timeout).

Then the client needs to implement the processing as follows:

Dequeue message 
if DequeueCount > threshold(2) then
    log bad message (put on poison queue) and delete it
    process message and delete it from the queue

The sequence of events is as follows:

  1. When the message is dequeued, the queue service increments the dequeue count by one, sets a invisibility timer on the message, and returns it to the client.
  2. The client reads the message and checks the dequeue count on the message.
  3. If the dequeue count is more than two, it means the message was dequeued twice in the past and its processing failed for some reason. It’s possible the message cannot be processed and will continue to show up on the queue. The message should be logged as a poison message and is often simply placed on a special poison message queue. This makes it possible for operations and development teams to analyze the cause of the failure.
  4. If the dequeue count is less than or equal to threshold value, then the message is processed and upon successful processing, deleted from the queue by the client

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.