I had a situation recently that an instance (not one I managed I hasten to add!) had started to grow in storage, to the point that it was at nearly 200% capacity. I was asked to look at it. Turned out that Plugin Tracing had been switched on, and a plugin was throwing errors. Unfortunately there were now so many records in the Log file that it wouldn’t open (Usual generic unhelpful SQL Error).
Taking a quick look at organisation insights confirmed that this was consuming nearly 60gb of storage, and a quick record count (Using Record Counter in XrmToolBox) indication there was approaching 10,000,000 records.
Normally I would use the standard Bulk Delete option, however for a couple of reasons I didn’t think this was a viable option:
In the end I decided to go back to trusty old SSIS, and (again) the lovely KingswaySoft solution for Dynamics 365.
The steps are essentially:
If you’ve used the SSIS Toolbox before, then you’ll know how to setup a connection.
To retrieve the data, I decided to go with FetchXML as I wanted to specify a date range, so the following was added to the Dynamics Source Data Flow:
<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false">
<entity name="plugintracelog">
<attribute name="plugintracelogid" />
<filter type="and">
<condition attribute="createdon" operator="on-or-before" value="2018-11-20" />
</filter>
</entity>
</fetch>So the Fetch above would pull back any log files created before 20/11/2018.
Once this is done, connect the Source to the destination, and open up the Destination Module
Add your connection in Connection Manager.
In Action, choose Delete.
In Destination Entity, choose the Plugintracelog entity
I enabled multithread, and chose 32 threads.
Once that is all done, Save, Build and Run.
So I don’t want to get in to the whole GDPR discussion in this article, but with GDPR in mind, I think it’s important, and potentially reduces risk, to anonymise data in a non production environment.
Again, there are plenty of articles on whether it should be a process of anonymising or pseudonymisation, but for the purpose of this article, I am going to go through the steps of using SSIS for the purpose of Anonymising Dynamics 365 Data.
Firstly, a few assumptions…..
You have Visual Studio and SSIS setup. I’m personally using VS 2015 Pro Update 3, SSDT.
You will also need a couple of essential Dynamics tools from the lovely people at KingswaySoft
SSIS Integration Toolkit for Microsoft Dynamics 365
They provide a developer license model to get you started.
Once you have everything downloaded and setup, then you’re ready to go.
Create a new SSIS Project
In Connection Manager, create a new connection
Next, drag over the following components:
In the Dynamics Source Component, Enter the Connection you just created, the Entity (in this case Lead).
Click on the columns section, and choose which fields from Dynamics you want to anonymise…… In this case I chose things like FirstName, LastName, Email etc.
Connect the Dynamics CRM Source component to the Data Anonymiser component by dragging the connector.
Open the Data Anonymiser Component.
Here you will see the fields/columns you have decided to anonymise, and the option to choose the data type
So for example the email address will be anonymised with data of email type
Do this for all your fields/columns.
Once this is done, open the CRM Destination Component
Next click on the columns section and map the source to the destination
So to summarize….
Once this is all done, save your solution and run it:
So in the instance above 274 Lead records were anonymised.
I hope you found this quick tutorial helpful.
