When Azure Diagnostics just won’t collect your data!!!

Recently realized that while we had happily implemented tracing and diagnostics in our applications , tools like Cerebrata , Azure MMC etc were just not showing any trace data.. We were doing everything by the book and even re-opened the training kit but the trace just won’t get collected. After A LOT of investigation and googling/binging it comes to light that the most optimized way of collecting diagnostics changed with Azure SDK 1.3 and further with 1.4.

If you are victim of “No WADLogsTable “ being generated try the below snippet. It just might resolve your woes

public override bool OnStart()
{
DiagnosticMonitorTraceListener tmpListener = new DiagnosticMonitorTraceListener();
System.Diagnostics.Trace.Listeners.Add(tmpListener);
string wadConnectionString = "Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString";
CloudStorageAccount storageAccount =
CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue(wadConnectionString));

RoleInstanceDiagnosticManager roleInstanceDiagnosticManager =
storageAccount.CreateRoleInstanceDiagnosticManager(RoleEnvironment.DeploymentId,
RoleEnvironment.CurrentRoleInstance.Role.Name, RoleEnvironment.CurrentRoleInstance.Id);

DiagnosticMonitorConfiguration config = roleInstanceDiagnosticManager.GetCurrentConfiguration();
config.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1D);
config.Logs.ScheduledTransferLogLevelFilter = LogLevel.Undefined;
config.DiagnosticInfrastructureLogs.ScheduledTransferLogLevelFilter = LogLevel.Warning;
config.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1D);
config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1D);

roleInstanceDiagnosticManager.SetCurrentConfiguration(config);
System.Diagnostics.Trace.TraceInformation("Diagnostics Running");

return base.OnStart();
}

Do let us know if this helped!!

Cennest!

Azure Scalability:- Use “Queues” as your bridges..

I feel the importance of Queues as a communication medium between Web and worker roles is quite a bit under-glorified…apart from acting as reliable messengers , if implemented properly Queues pretty much hold the key to your application being Extensible and Scalable

Windows Azure Queue allows decoupling of different parts of a cloud application, enabling cloud applications to be easily built with different technologies and easily scale with traffic needs.

This above definition is full of some very important keywords..decoupling.…different technologies…. scale with traffic needs…all very important to make a scalable and extensible application..

Lets see how Queues help in all these aspects.

1.  Scalability:-

  • Observing the queue length and tuning the number of backend nodes accordingly, applications can effectively scale smoothly accordingly to the amount of traffic. A growing queue length or an almost always zero queue length is an indication of how loaded your worker roles are and you might want to scale them up or down accordingly
  • The use of queues decouples different parts of the application, making it easier to scale different parts of the application independently. This allows the number of web and worker roles to be adjusted independently without affecting the application logic.
  • Separate queues can be used for work items of different priorities and/or different weights, and separate pools of backend servers can process these different queues. In this way, the application can allocate appropriate resources (e.g. in term of the number of servers) in each pool, thereby efficiently use the available resources to meet the traffic needs of different characteristics

  2.     Extensibility:-

  • Different technologies and programming language can be used to implement different parts of the system with maximum flexibility. For example, the component on one side of the queue can be written in .NET framework , and the other component can be written in Python.

3.   Decoupling:-

  • Changes within a component are transparent to the rest of the system. For example, a component can be re-written using a totally different technology or programming language, and the system still works seamlessly without changing the other components, since the components are decoupled using queues.  The old implementation and the new implement can run on different servers at the same time, and process work items off the same queue. All these are transparent to the other components in the application

But to be able to achieve all these benefits you need to make sure your queues are implemented in the most optimal way . There are a lot of best practices around Queue implementation such as Invisibility Time, Delete queue messages, Worker role instance manipulation etc…more on that some other time…meanwhile if you want more inputs on “Queueing your way to scalability” drop us a a note at Cennesttech@hotmail.com..

Until then!

Cennest

Bulb Flash:- Azure Message Queues-Passing objects

Azure queue takes either an array of bytes or a string as an input. While working on a web role- worker role interaction , i needed to pass the entire TableServiceEntity object(basically the Azure table datamodel) to the worker role.

I needed the partition key, the row key as well as some other attributes of the database entity in the worker role to do some processing…the best way out was XML Serialization!!

//serialize the tableserviceentity object in your webrole

Suppose your TableServiceEntity class is called XYZDataModel

//Deserialize it in your worker role

And now you have your XYZDataModel entity recreated in your worker role!!..Hope this helps you..We certainly will try to remember this piece of code!!

Thanks

Cennest

Programming Table Storage in Azure…Choosing your Keys!!

Azure holds a lot of promise for applications with

1. Spikes in usage

2. Requirement for low set up cost

3.Huge storage requirements

Out of these 3 major benefits, most applications will exploit the 1st and the 3rd potential of Azure!..While Spikes in Usage is more related to how the application is architected etc(we’ll talk about it in another post), huge storage requirements is a pretty interesting topic…the potential here is enormous..

You want to learn more on Silverlight 4, won’t it be cool if you could just see what articles people have read and which ones they found useful and why?? I am sure it will help you make more out of your time by helping you weed out the “not so useful” articles….but who would take the onus of storing such HUGE data?? Well something like Table Storage in Azure could be the answer to this..it is a scalable, extensible and inexpensive storage medium to store just this type of data…

While just having fun with Azure, we made this small app which allows a user add a weblink onto the Azure table storage…

image

he can then also search the entire table for weblinks of his interest based on categories..

image

One can take this forward to search by submitter, or Category+ rating etc etc..

The entire application is uploaded here…its a rough sample (so no commenting etc etc )but it will give you a idea..if you have any issues with it…its cennesttech@hotmail.com!!!

There are lots of samples out there so i don’t want to focus on “how to program” this….i am more interested in the most important aspect of using the Table Storage in Azure….How to select you Keys!!(Partition + Row Key)

Basically your partition key should be the most commonly used filtering criteria for your application….so basically you need to realize the main purpose of the application and in what way it is going to be used, what kind of queries are going to be most frequent…one tip here is that the you should be able to use your partition key in the where clause of almost all your queries..

Lets see what kind of querying makes sense for this simple application

1. Get me all weblinks in CategoryX

2. Get me all weblinks in CategoryX with Rating Y

3. Get me all weblinks submitted by User T

4. Get me all weblinks submitted by User T in Category X

etc, etc

If you notice most querying is going to involve the “Category”, so we should be able to use Category in the where clause!!….so the partition key becomes category!!

You must use Category as a condition in most of your queries

image

Now for the Row Keythe rule here says that if there are 2 frequently used keys, use the most freq as the partition key and the other one as the Row Key…so if you see the querying, you will be tempted to use the “SubmittedBy” as the row key…but there is a problem…PartitionKey+RowKey should be unique for every row in the table…so if you use SubmittedBy as the Row key, a user will be able to submit a weblink for a category only once…makes no sense!!

For applications such as this…it makes sense to use Guid.NewGuid as the row key to allow multiple entries and still maintain uniqueness…

Like i said, let me know if you want the source code…will be happy to load it somewhere for you!!…Azure table storage has huge potential as long as the keys are selected properly!!!

Thanks

Cennest!!

Bulb Flash:- Remember the restrictions in Azure!!

Was collating all sorts of size restrictions for tables, blobs etc but found this really useful post by Cerebrata which has the exact info i wanted to collate!!

Another useful blog by them is on the differences between Dev storage and cloud storage…will help prevent frustrating moments while going uphill the learning curve!!

Thanks Cerebrata!!

Cennest!