HELP! My Application Insights EXPLODES!

At QBS Group we are promoting the use of Application Insights. We do that not because we are nerds and love new technology, but also because of simple home economics.

Before the introduction of Application Insights last year, we would often see long-running support tickets between our partners and Microsoft to troubleshoot problems with Business Central. Application Insights changed that almost overnight by giving partners and even customers the option to see where problems are caused.

Although we did not do any exact math, we feel it is safe to say partners can save hundreds, if not thousands of euro’s by using Application Insights.

What exactly does it cost and how do I monitor it?

But what does Application Insights cost per tenant or per ISV Solution in AppSource and do the economics still apply?

Running an Application Insights resource in Azure is not free of charge. The first 5GB per month is paid for by Microsoft with a retention period of 90 days. This sounds like it should suffice and in most cases it does. However, some of our partners reported to us that they had examples where they had much more data than that which resulted in a cost of tens, even hundreds of euros per month to host Application Insights.

Together with Microsoft, we investigated the issues, and we feel we should share these experiences with you.

Executive summary

As an executive summary, the conclusion is that having an Application Insights instance that starts charging you for a Business Central tenants telemetry is a warning of problems as such. An average tenant should not produce enough telemetry to cause the invoicing process in Azure to start.

So, what can trigger Application Insights to explode, what can you do about it and lastly, how can you avoid having to pay hundreds of euros for an overflow of telemetry cause by, for example, an App from AppSource with programming errors.

We investigated a few instances that we have access to which were large. A few tenant instances, an AppSource ISV instance and an Embed App instance.

How to analyze the cost

To analyze which events cause the problem we used this KQL query that Microsoft provided and that they shared on GitHub now too.

traces

| where timestamp > ago(30d)

| project eventId = tostring( customDimensions.eventId )

| summarize count() by eventId

For now, we will ignore AL000EAV because Microsoft confirmed it as a bug. I will get back to that at the end of the blog.

We need to focus on the other events, RT0004, RT0019, RT0009 and RT0020. This means (Pre) Open Company and Web Service calls.

In fact, RT0019 is called 160.000 times which is the equivalent of once per second. In other words, someone is polling something, and the solution is probably to implement an event driven system like Azure Event Grid. The culprit here was a partner customization.

Let’s look at another example.

A whopping 67 million counts into a Web Service from Business Central. In fact, this happened on a few days with the 30 days period generating 50GB of telemetry on one single day generating a 300+ euro invoice for this ISV.

In this case we narrowed the problem to Azure Blob Storage called millions of times to ping if a file exists.

What does this cost exactly?

Don’t kill your own system

The examples we found that were overflowing Application Insights can all be related to software that behaves in a polling mechanism used for interfaces leading to excessive amounts of web service calls both inbound and outbound.

We believe Business Central is not designed for this and with the Azure world, better options exist that are event-driven rather than timer driven. Logic Apps and Azure Event Grid are just two examples that we will share more examples of in the near future.

Avoid large bills

Lastly, I promised to explain how to avoid getting large invoices for Application Insights. Your customers may install an App from AppSource that results in a cost on your tenant without you knowing.

Also, let’s be honest, a bug by Microsoft may also cause a lot of records in your systems and it would be fair not to have to pay the same company for that.

The solution is to cap the amount of data and setup email alerts in a case large number of events are created in the system.

For a Business Central tenant, we recommend capping on 50MB per day and raising an alert by email if one event id is created more than 5000 times per day. Of course, there can be exceptions to this rule.

ISV’s can limit to 1GB per day to start and let the system grow over time if the number of (hopefully paying) customers grows. We recommend ISV’s to look at Application Insights on a daily bases and monitor delta’s rather than just large numbers. You can also raise alerts if you see a new tenant id coming in which probably means someone installed your solution from AppSource.

You can read how to set this up on Microsoft Docs or by registering to the QBS Academy Application Insights Class.

Useful links:

Microsoft  BCTech AppInsights

Microsoft BCTech AppInsights FAQ