måndag 20 maj 2019

Round up of data consumption in Microsoft Dynamics 365 Customer Engagement



I thought I should somehow round up the “Minimal copy” blogs with some sort of conclusions or rather recap perhaps. The base of this blog is that a minimal copy of a production instance wound up being 4.3GB which later was downsized with roughly 3GB.

The base of this text is “What are we paying for” when a subscription of Microsoft Dynamics 365 Customer Engagement is started. This has actually just changed but there are still some questions from my side. As of now, or when your subscription renews, you will have 10GB of database capacity, 20GB of file capacity and 2GB of log capacity per subscription.
The change is to begin with that the “attachments to notes or emails in Dynamics 365 Customer Engagement applications and PowerApps” (from the license document) no longer is counted as database storage, which is a good thing. I don’t know how many customers have wanted to delete attachments and there’s even a solution moving attachments to Sharepoint, which still is a good idea since Sharepoint is better at storing documents. 

The next thing that’s nice is that logs are counted as a separate store. I can’t say from the top of my head how much 2GB of logs are worth since it depends on how much auditing you are doing, the possible downside of this is that you don’t get any extra log storage from buying more licenses of Dynamics 365 Customer Engagement (I will address this later).
Sometimes this is how I feel
Now, database storage. 10GB of database storage, what does that mean? Again from the licensing document: “Common Data Service Database Capacity stores and manages entity definitions and data. The tenant for Dynamics 365 Customer Engagement plan and application subscriptions includes by default 10GB for database capacity.”

Ok, entity definitions and data. This means we are paying for the data we put into the system and for the schema. Not only the customizations we do but the standard schema as well. This might very well be reasonable but I don’t think this is what most customers think they’re paying for so it’s a good thing to know if you’re a partner and also good to know if you’re a customer.

This is where things gets interesting to me because a completely out of the box new instance of Dynamics 365 v9 on premise (ok latin-something collation since finnish_swedish is not working atm) is roughly 1.1GB compared to a Dynamics 365 v8.2 which is around 0.6GB. This 1GB storage of an empty instance is also true when we have Dynamics 365 Customer Engagement online, which I actually thought was a bit much when discussing it with the support but it turns out that the vanilla-no-data instance has doubled in size during the last version.  So when we look at 10GB of data per subscription, more than 1GB of that is taken per instance we’re having on it. For example, if we would have 1 production instance and 2 sandboxes (1 development and 1 test), 3GB of data would be consumed by just the instances. This is something that’s good to know at least.

Moving on with the licensing experience (ok, I don’t know if it’s still called this but it’s a too good a name to not use. It was called this during a presentation of Dynamics CRM 2013) you will get an additional 0,25GB of database storage and 2GB of file storage per Enterprise license. No log storage is added by having licenses. Extra storage is bought by the GB per type and I haven’t been able to find any price for this but that might be a case of bad Google-fu on my part (sorry but Bing-fu doesn’t sound as catchy).

When I was starting to dig into this I was able to download a report from the Power Platform Admin Center (one of three or four admin centers at the moment) which showed in which tables data was consumed. That report has since been removed after Microsoft changed the storage thingy to a new name and a new portal which doesn’t show as much information as it used to, yet at least. These portals and information flows tend to shift over time. The result as of 2019-05-20 is that you can’t tell where your data is consumed. What didn’t show in that report though was how much data that was consumed by “not your data” i.e. the system so a completely empty instance will eat some 1GB of data storage as I wrote earlier. And with that report being removed, I can’t tell what a completely empty instance will show as consumed for contacts for example.

Conclusions 

This all boils down to that there’s no real way of telling what you actually consume and we have to rely on Microsoft telling us what is actually in our instances which in this case wasn’t entirely true and I had to push back rather hard to make them believe that 4.3GB of data for an empty instance is not reasonable. This is easy to think that something’s wrong in an instance that you would think is empty but when you have an instance filled with data it’s quite the opposite.
In this case we made a minimal copy from production to an empty instance and started to fill it with data from source systems thinking we would use that instance as the new production so we cleared the “old production”. After a while we made a choice to redo the migration and did a minimal copy from the just filled instance to the “old  production” and when that was done, the “sort-of-.reliable-source” of this was erased. So, for most cases you risk getting this issue in the instances you set up from another instance. And in most cases that shouldn’t be all your instances, even if it’s not unthinkable it’s more than one.
The data usage per table report that has been removed will hopefully be re-introduced shortly.




Rickard Norström
Developer at CRM-Konsulterna
www.crmkonsulterna.se 

onsdag 15 maj 2019

Minimal copy follow up.

I started a case with the minimal copy issue as stated earlier. About a week later after some rather heated discussion with the support a bit over 3GB of each instance was removed.

It was at times interesting to say the least when talking to the support crew. That being said, I have no grudge whatsoever with the people I was talking to since it wasn't their job that I was debating but rather the back end.

The first answer I got was that it was data from the instance I was copying to, since that wasn't empty when I started, there wasn't any real explanaition to why I couldn't access any of that data to remove it though.

Answer number two was that it was metadata and customizations, when I asked why the development instance was a mere 1.3GB that didn't really give any answers either. I think that the 1.3GB is a bit on the heavy side too considering that a somewhat filled Dynamics 365 Customer Engagement 8.2 on premise database is around 0.8GB so I was asking how come an "empty" instance is almost twice the size of an on prem one too. After an hour long screen sharing session where I did a reset of the test instance before I did a new minimal copy from production which gave the same result as before, which I assume removed the idea that it was old data taking up the space, the case was still open.

At this time the power platform admin center wasn't available any longer for this instance, and I don't think it's back up yet either so I couldn't see which tables were creating the most impact any longer. the Dynamics admin center told me that the size was still on the large side though.

After little more than a week the size was considerably smaller, a little over 3GB per instance was shaved and the case was closed. I didn't get any info on what data was removed more than that scripts had been run on the backend to recalculate the size of the instances.

The moral of this is that it's a good idea to have a look at your instances after a minimal copy to see if they are around the 1GB mark unless you have a shitload of customizations of course. Sometimes the answers given by the support team isn't correct unfortunately and it helps having had a look at how the databases behave. The thing that annoys me most in all of this is that I got answers two times that didn't really told the truth and that Microsoft charges by the GB, in this case 6 of them, which is just air.

Rickard Norström
Developer at CRM-Konsulterna
www.crmkonsulterna.se