Azure CloudQueue, Get and Set Json using Newtonsoft.Json and Microsoft.WindowsAzure.Storage

The Microsoft Azure Cloud have a nice simple Queue mechanism in their Azure Storage where you can store up to 64 kb of data for up to 7 days. The queue is very easy to set up and very easy to work with when using C#.

The Azure Storage Queue stores simple text strings, but with Newtonsoft.Json you can serialize and deserialize Json messages and by that create a Json based message queue system.

To use the Azure Storage Queue you need to set up an account. Click here to see how an account is made. Creating the account creates a key that can be concatenated to a connection string of the following format:

DefaultEndpointsProtocol=https;
AccountName=YOURACCOUNTNAME;
AccountKey=A_VERY_LONG_ACCOUNT_KEY

With this connection string you can access the queues you made in the system.

STEP 1: CREATE A GENERIC REPOSITORY TO GET AND SET OBJECTS

This repository allows you to get and set messages to and from the Queue.

using System.Collections.Generic;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;
using Newtonsoft.Json;

namespace MyNamespace
{
  public class AzureQueueStorageRepository
  {
    private readonly CloudQueue _queue;

    /// Create an instance of [see cref="AzureQueueStorageRepository"]½
    /// <param name="queueName">Name of queue</param>
    /// <param name="connectionString">Azure Connectionstring</param>
    public AzureQueueStorageRepository(string queueName, string connectionString)
    {
      Assert.ArgumentNotNullOrEmpty(queueName, "queueName");
      
      CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
      CloudQueueClient client = storageAccount.CreateCloudQueueClient();
      _queue = client.GetQueueReference(queueName);
      _queue.CreateIfNotExists();
    }

    /// Retrieves number of messages from queue. Messages are removed from queue when retrieved
    /// <param name="count">Number of messages to retrieve</param>
    public IEnumerable<CloudQueueMessage> GetMessages(int count)
    {
      int batchRetrieveCount = 32; //max batchsize
      var retrievedMessages = new List<CloudQueueMessage>();
      do
      {
        if (count < 32)
          batchRetrieveCount = count;
        IEnumerable<CloudQueueMessage> receivedBatch = _queue.GetMessages(batchRetrieveCount, TimeSpan.FromHours(48));
        retrievedMessages.AddRange(receivedBatch);
        DeleteBatch(receivedBatch);
      } while ((count -= batchRetrieveCount) > 0);
      return retrievedMessages;
    }

    /// Peek one message from the queue. The message is not removed from queue, nor marked as invisible.
    /// <returns>Peeked message</returns>
    public CloudQueueMessage Peek()
    {
      return _queue.GetMessage(TimeSpan.MinValue);
    }

    ///
    /// Delete one message from the queue
    /// <param name="message">Message to delete</param>
    public void Delete(CloudQueueMessage message)
    {
      _queue.DeleteMessageAsync(message);
    }

    /// Add message to queue
    /// <param name="messageObject">Message to add</param>
    public void AddMessage(object messageObject)
    {
      string serializedMessage = JsonConvert.SerializeObject(messageObject);
      CloudQueueMessage cloudQueueMessage = new CloudQueueMessage(serializedMessage);
      _queue.AddMessageAsync(cloudQueueMessage);
    }

    private void DeleteBatch(IEnumerable<CloudQueueMessage> batch)
    {
      foreach (CloudQueueMessage message in batch)
      {
        _queue.DeleteMessageAsync(message);
      }
    }
  }
}

The repository itself does not know which Json messages you get or set, which is why the GetMessages() returns the Microsoft.WindowsAzure.Storage.Queue.CloudQueueMessage and the AddMessage() inserts an untyped object.

STEP 2: HOW TO ADD AN OBJECT TO THE QUEUE AS A JSON STRING

To add an object to the Queue as a Json message you need to have an object, serialize it to Json and then store this object. Here is a sample object than can be serialized to Json:

using Newtonsoft.Json;

namespace MyNamespace
{
  public class MyMessage
  {
    [JsonProperty("propertyString", Required = Required.Always)]
    public string PropertyString { get; set; }

    [JsonProperty("propertyInt", Required = Required.Always)]
    public int propertyInt { get; set; }
  }
}

To store this message in the Azure Storage Queue you can do the following:

var message = new Mymessage();
message.PropertyString = "Hello World";
message.propertyInt = "2015";

var rep = new AzureQueueStorageRepository("queuename", "connectionstring");
rep.AddMessage(message);

STEP3: HOW TO GET AN JSON STRING FROM THE QUEUE AND DESERIALZE IT TO AN OBJECT

To get the message from Azure Storage Queue you can do the following:

The repository returns a CloudQueueMessage that contains the Json as a text string. This string needs to be serialized into an object. To help with this I have created a helper method that I can call for all my deserialization needs:

/// Deserializes JSON into object of type T.
/// <typeparam name="T">object of type T to deserialize to</typeparam>
/// <param name="json">JSON string</param>
/// <param name="ignoreMissingMembersInObject">If [true] the number of fields in JSON does not have to match the number of properties in object T</param>
/// <returns>object of type T</returns>
public T Deserialize<T>(string json, bool ignoreMissingMembersInObject) where T : class
{
  T deserializedObject;
  try
  {
	MissingMemberHandling missingMemberHandling = MissingMemberHandling.Error;
	if (ignoreMissingMembersInObject)
	  missingMemberHandling = MissingMemberHandling.Ignore;
	deserializedObject = JsonConvert.DeserializeObject<T>(json, new JsonSerializerSettings
	{
	  MissingMemberHandling = missingMemberHandling,
	});
  }
  catch (JsonSerializationException)
  {
	return null;
  }
  return deserializedObject;
}

With this helper class, deserialization is easy:

var rep = new AzureQueueStorageRepository("queuename", "connectionstring");
IEnumerable<CloudQueueMessage> messages = azureQueueRepository.GetMessages(1);
foreach (CloudQueueMessage message in messages)
{
  string jsonString = message.AsString;
  MyMessage myMessage = Deserialize<MyMessage>(jsonString, true);
  
  // do with the object whatever you need

}

MORE TO READ:

• How to use Queue storage from .NET by Microsoft
• About Azure storage accounts by Microsoft
• Azure Queues and Service Bus queues – compared and contrasted by Microsoft
• Json.NET by Newtonsoft