Tag Archive SSIS

ByRob Peledie

Using SSIS for Bulk Data Deletion in Dynamics 365

I had a situation recently that an instance (not one I managed I hasten to add!) had started to grow in storage, to the point that it was at nearly 200% capacity. I was asked to look at it. Turned out that Plugin Tracing had been switched on, and a plugin was throwing errors. Unfortunately there were now so many records in the Log file that it wouldn’t open (Usual generic unhelpful SQL Error).

Taking a quick look at organisation insights confirmed that this was consuming nearly 60gb of storage, and a quick record count (Using Record Counter in XrmToolBox) indication there was approaching 10,000,000 records.

Normally I would use the standard Bulk Delete option, however for a couple of reasons I didn’t think this was a viable option:

  1. It would take too long
  2. It kept falling over after 1000 deletes (probably due to the number of records, but I didn’t get to the bottom of it)

In the end I decided to go back to trusty old SSIS, and (again) the lovely KingswaySoft solution for Dynamics 365.

The steps are essentially:

  1. Create a CRM connection
  2. Retrieve what you want to delete (Using Dynamics Data Source Component)
  3. Delete

If you’ve used the SSIS Toolbox before, then you’ll know how to setup a connection.

To retrieve the data, I decided to go with FetchXML as I wanted to specify a date range, so the following was added to the Dynamics Source Data Flow:

<fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false">
  <entity name="plugintracelog">
    <attribute name="plugintracelogid" />
    <filter type="and">
      <condition attribute="createdon" operator="on-or-before" value="2018-11-20" />

So the Fetch above would pull back any log files created before 20/11/2018.

Once this is done, connect the Source to the destination, and open up the Destination Module

Add your connection in Connection Manager.
In Action, choose Delete.
In Destination Entity, choose the Plugintracelog entity

I enabled multithread, and chose 32 threads.

Once that is all done, Save, Build and Run.

ByRob Peledie

Anonymising Data in Dynamics 365 with SSIS

So I don’t want to get in to the whole GDPR discussion in this article, but with GDPR in mind, I think it’s important, and potentially reduces risk, to anonymise data in a non production environment.

Again, there are plenty of articles on whether it should be a process of anonymising or pseudonymisation, but for the purpose of this article, I am going to go through the steps of using SSIS for the purpose of Anonymising Dynamics 365 Data.

Firstly, a few assumptions…..

You have Visual Studio and SSIS setup. I’m personally using VS 2015 Pro Update 3, SSDT.

You will also need a couple of essential Dynamics tools from the lovely people at KingswaySoft

SSIS Integration Toolkit for Microsoft Dynamics 365

SSIS Productivity Pack

They provide a developer license model to get you started.

Once you have everything downloaded and setup, then you’re ready to go.

Create a new SSIS Project

In Connection Manager, create a new connection








Next, drag over the following components:

  • Dynamics CRM Source
  • Data Anonymizer
  • Dynamics CRM Destination








In the Dynamics Source Component, Enter the Connection you just created, the Entity (in this case Lead).







Click on the columns section, and choose which fields from Dynamics you want to anonymise…… In this case I chose things like FirstName, LastName, Email etc.







Connect the Dynamics CRM Source component to the Data Anonymiser component by dragging the connector.

Open the Data Anonymiser Component.






Here you will see the fields/columns you have decided to anonymise, and the option to choose the data type

So for example the email address will be anonymised with data of email type





Do this for all your fields/columns.

Once this is done, open the CRM Destination Component







Next click on the columns section and map the source to the destination







So to summarize….

  • CRM Connection
  • Choose Source entity and Fields
  • Choose Anonymise options
  • Set Destination Mapping

Once this is all done, save your solution and run it:









So in the instance above 274 Lead records were anonymised.

I hope you found this quick tutorial helpful.