AX 2012 – Data Migration Framework: Usability to date

I hope everyone is having a great week thus far! Being head strong into summer and WPC12 behind us all, it’s time to take the biggest year in Microsoft Product history and unleash an unprecedented amount of value on the world. Microsoft Dynamics AX customers will benefit from all the great products, from SQL Server 2012, Windows Server 2012 and Visual Studio 2012. Let us also not forget about the value proposition that Windows 8 will bring for Phone, Tablet, PC and more!

With that said, today I wanted to wrap up my review of the recently released Microsoft Dynamics AX Data Migration Framework. I started this coverage early this week, with the following two post covering the release and step-by-step setup.:

• AX 2012 - Data Migration Framework (Beta) Released
• AX 2012 – Data Migration Framework: Step-by-step setup

With this now understood, I spent some time looking into the current usability of this beta release for the Data Migration Framework.



Lets point out again, that this is the beta release and therefore there will be bugs, issues and it’s not a complete offering. With that, when we start to look at the use of this framework, we must start with the types of sources we can work from. We do this from the Data Migration Framework > Setup > Source Data Formats.



I want to point out that as you open the form, we can see there is a Type field. In looking at this closer however we can’t change this from the type of File. You can see this type in the above referenced screen shot. Further inspection of this form, and it’s table DMFDataSource and we see that the field type does not allow edit. This is shown in the screen shot below.



In looking at the Base Enumeration that this field represents, being that of DMFSourceType, we do find out that there are two other types.



These types are of Ax and ODBC. We can see that there are plans then for allowing both ODBC source types, as well as enabling AX-to-AX, source-to-target usage of this framework. Just not yet! Again this is a beta, which I would think will be getting a refresh pretty soon to enable these other type source types.

Moving along, since this is file based on source types for the Data Migration Framework, out-of-the-box usage for the framework is limited. It’s powerful, and has a lot of great possible value, specifically when the AX and ODBC types are fully added. Further the framework does work well with the file sources, as I’ve tested and imported products via. If your source data is in flat files, or you can export to flat files, then you can start getting a lot of use from the framework now.



Just to show that Microsoft does plan on really helping to enable this framework to have a multitude of sources, you can see proof in the naming of .Net assemblies that come as part of the install. Specifically the below screen shot of the ‘Microsoft.Dynamics.AX.DMF.SSISHelper’ .Net assembly reference.



In looking for how this is currently used in the X++ code that is delivered along with the install of the model file, we find the ‘DMFGenerateSSISPackage’ class. As you can see from the below screen shots however, this is only targeted for file based sources, for now.







With all of this stated, if you have more complex data import needs, or your already well under way with the concept of using services to import data for Microsoft Dynamics AX 2012 there still is value is the data migration framework for your needs.



If we take the above screen shot, we can see the installed DMFProductEntity table. This is the staging table for products. This table, along with other staging tables, can be targets of such services or import targets for more complex importing scope.



Finally on this topic around value, there is a ‘Create a custom entity for migration’ wizard that can be called to target any out-of-the-box entities or custom ones that you may have. This is shown in the screen shot above. Further the screen shot below shows the Wizard in action with what table should be the target of a custom migration entity.



With everything said, it’s a great start to helping fill a need. As I mentioned before we have been using this approach for over 2+ years now on Microsoft Dynamics AX 2012 projects. What this helps with is enabling a true framework, that is repeatable and everyone can build from. Like all things in AX, you can customize this, and for your custom entities you most likely will.

Well that’s all for this post, and my current coverage of the newly release Microsoft Dynamics AX Data Migration Framework. I will come back to this topic, at a later date when Microsoft has updates that are worth sharing around this topic. Further, starting tomorrow I will get back to my regularly scheduled topics around the Cloud, BI, EP Development – and our theme of Creating a System of Engagement with Microsoft Dynamics AX 2012. Till Next Time!
Follow Me @:
    

"Visit the Dynamics AX Community Page today!"

AX 2012 – Data Migration Framework: Step-by-step setup

Recently I wrote about the release on InformationSource, of the Microsoft Dynamics AX 2012 Data Migration Framework. In that post, I mentioned that I would continue to cover this release, including walking through the process of setup, configuration and use. In this post I plan to start that coverage as promised, with a step-by-step setup of the Data Migration Framework.



Visit Hillstar Business Intelligence (www.HillstarBI.com) in order to truly unlock your data trapped in your Microsoft Dynamics investment. With our value driven business intelligence strategy Hillstar help you transform into a data informed company.


The idea with this post, is to walk you through the screens, points of interest, through the full compile and finally to the point of configuration. At that point, we will continue later with a new post on the configuration and use of the Data Migration Framework.

Now that we understand the purpose, and scope of this post, lets get started. First off, of course you need to download the Data Migration Framework from InformationSource. After having downloaded the Data Migration Framework, the next step is to extract the contents of the Zip file.

At this point you will want to launch the Setup for the Data Migration Framework.



Moving from here a decision must be made on what is to be installed.



Since I’m making the choice to install both Server and Client on a development instance of AX 2012 I will leave both checked. It’s worth to note the message contained within the first dialog box, around the fact that the server part must be installed on an AOS.

Moving along we must select the path in which the setup will install the application files to.



After this, you should reach the final screen, indicating that the setup program has completed it’s task.



When clicking on the finish button, the installation log file opens up for you. It’s worth scanning through to see if there are any errors.



Now that we have completed the install, we need to double check that we have the Data Migration Framework bits ready for our use. Do this by browsing to the location in which you had chosen previously. In doing so, you should see a folder with similar contents as shown below.



Now that we have verified that install completed successfully, lets move towards a point in the install documentation on TechNet.



Specifically there is a warning, that states: “Because the staging environment is highly normalized and may require significant processing bandwidth, we recommend that you increase the Maximum buffer size setting for your environment while you migrate data. Use the Server configuration utility to set the value.” Lets take the advice of Microsoft and do just that.

This means you need to launch the server configuration utility, go to the database tuning tab, and change the buffer size from 24, to say 48.







You will notice that in making such a change Microsoft warns of doing so. Since we are doing this on purpose, we can ignore this warning. However it’s valid to point out that after your done with the data migration framework, it’s best to change this setting back to 24. Further, you might find that 48 is still not big enough and you will need to increase further.

After doing this, and clicking apply you should take the option of allowing the configuration utility to restart the Microsoft Dynamics AX 2012 AOS instance.



Now we have to move to importing the model file that came as part of the install bits for this offering. We perform this install either by using the PowerShell command console, or the AXUtil application. I’ve made the choice of using the AXUtil application.



Finally after a successful install of the model file, to your target instance of Microsoft Dynamics AX 2012, you should launch AX. In doing so, it will detect a new model as been installed, and ask how you wish proceed. For this, we want to make the choice of ‘Compile and synchronize’ and then click ok.



After the successful completion of the full compile and synchronization of the database, next action to take is a full CIL compile. This should take roughly 15 +/- minutes to complete. After completing this step, your instance of AX 2012 has the newly released Data Migration Framework deployed and setup. You should see the something similar to the following when done.



Now that we have the Data Migration Framework deployed and setup, the next steps goes towards configuration. This is where creating entity maps, and understanding source through to target needs great focus, clarity and I can stress enough human workflow.

Well that’s all for this post. I hope you find it useful and helpful in speeding up your adoption and use of the Microsoft Dynamics AX Data Migration Framework. Keep in mind that this is still in beta, and there will be issues. So keep a watch on InformationSource and further this blog for updates. Till Next Time!


Visit Hillstar Business Intelligence (www.HillstarBI.com) in order to truly unlock your data trapped in your Microsoft Dynamics investment. With our value driven business intelligence strategy Hillstar help you transform into a data informed company.


Follow Me @:
    

"Visit the Dynamics AX Community Page today!"

AX 2012 - Data Migration Framework (Beta) Released

I hope everyone is doing great this fine Dynamics Day! I thought I would warm back up the writing presses, if you will, with some exciting news. Microsoft has released to InformationSource the Beta release of the Microsoft Dynamics AX 2012 Data Migration Framework



The above is from the Data Migration User's guide, which you can find at the following location.: Data Migration Framework User Guide [AX 2012]

Further you can download the beta release for this new framework, located at InformationSource from the following link: Microsoft Dynamics AX 2012 Data Migration Framework (Beta) Download. Keep in mind that you do need PartnerSource or CustomerSource access for this download. Further please keep in mind this is Beta.

The approach that this data migration framework is something that we at IDB Solutions have been using for now two years working with Microsoft Dynamics AX 2012. The basics are simple, you de-normalize the datasets that make up the Data model in AX 2012 with targeted staging tables. Then you have X++ code that work against the staging tables, to fill the normalized, and sometimes super normalized, datasets of AX 2012.



The configuration aspect, or process for this framework is spelled out below. This was also taking from the above referenced user's guide.:

• 1) Determine entities in source system
• 2) Define the source data format
• 3)Identify entities to place in processing group
• 4) Define the processing group
• 5) Source to Staging (Entity Map)
• 6) Staging to Target (Entity Map)

Further there is also a Model file that needs to be installed into your target instance of AX 2012. You can do this, with the following example - based on the AXUtil application.: 'axutil import /file: "C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (CTP)\DataMigrationFramework.axmodel"'. You can find this as part of the above referenced user guide, in the install section.

I'm in the process of downloading this now for use on my clients AX 2012 projects. I plan on having a follow up post on my thoughts of the framework, ease of use, extending it and more.

That's all for this kick off post, and check back soon as I continue to cover this new framework, as well as more cloud stories, a continue focus on EP and of course BI on the brain!Till Next Time!
Follow Me @:
    

"Visit the Dynamics AX Community Page today!"

Planning for data import, export & migration with AX 2012

With every Microsoft Dynamics AX 2012 implementation, a very import topic, that involves a lot of planning, and technical work, is around the process of migrating data. This includes a lot of exporting, importing, transformation and so on.

Like all task relating to an AX 2012 implementation, these should be governed and defined based on functional business requirements and discovery. That is what should drive the choice being made, for how specific entities are imported, migrated or even entered into AX 2012.

To that end, and with this in mind, lets look at some new content on MSDN, specifically on this topic.: Plan data import, export, and migration [AX 2012]

From the article:
"This topic describes the tools and strategies to use when planning to import or export Microsoft Dynamics AX data. It describes how to plan to migrate data from one enterprise resource planning (ERP) system to another. Finally, it describes performance and security considerations for data import and export."

This is actually a very well thought out, and put together article, and I highly recommend it be used as a source when in reference to such task, when working in this area of a AX 2012 project.

With that said, there are some understandings that we need to circle back around to, to allow business requirements and customer needs to drive what is completed, in regard to such task.

One broad stroke statement that is not always applicable from this article is the following.:
"We[MSFT] recommend that you do not migrate transactional data or historical data to Microsoft Dynamics AX. Instead, close all open transactions before migrating, if it is possible. Maintain the database that contained your previous transactional data for reporting and compliance purposes."

This, as many of you senior consultants understand full well, is a great statement, and if it's possible really helps reduce the cost of data migration efforts in a project. However, it's hardly always true, and would say that the above statement is typically not what takes place for open transactions on any AX project.

Instead, what we try to push for, in the above statements place, is based on volume for open transactions. With that, if the volume is not to large, then a two birds, with one stone event can occur.

Instead of having to write data migration logic, this can be used as a training opportunity and get the open transactions into the new AX 2012 instance. There still can be an argument that even if there is a perceived large amount of volume to such data, that it's cheaper and better ROI to have more hands entering in the open transactions than writing migration code.

Why would that be? The reason is, any time spent, and therefore budget dollars burned, on the process of data migration is all throw away once the go live takes place. This means it looses it's value, as soon as the new AX 2012 instance is up and running.

So the trick, for adding the most value to such projects, is to perform the least complex, and spend the least amount of time possible in this area. It's not the the point that it should be overlooked, I'm not stating that at all. What I am saying, is that whatever the solution for migration is, it should be defined based on the understanding that the work will be throw away as soon as the go live takes place.

Now, on larger implementations, most likely all of the options that you see listed in the above document will be used. There is no single way to achieve importing all entities, instead, each should be defined, based on volume, complexity, and level of transformation that needs to take place. The process should be as fast as possible, and least amount of time and therefore budget dollars. Simple is king of value, and rules over it in this area with an iron fist.

Another point I would like to bring up, that is not listed in the above article, is around RapidStart Services and it's impact on data migration, specifically around reference and setup data. With the release of RapidStart Services for AX 2012, this too should be considered, weighted and then valued for helping create data entities, and for importing other entities via.

I will leave with this, always look for the most simplest approach when migrating data. Only be as complex as the requirements and customer constraints force you to be. Sometimes the most elegant solution, is the worst in value, when it comes to the task related to data import, export & migration.

Finally, I will reference back to this post in future articles, as I review some of the options in detail. Talk about the real world impact, by example, and give some food for thought to help better understand the value of the choices made on real projects.

That's all for now, till next time!



"Visit the Dynamics AX Community Page today!"

Microsoft Dynamics AX 2012 - A Dive into Services, Consuming Document Services

With the release of Microsoft Dynamics AX 2012, one of the topics I have been spending a lot of time talking about is around services. There is good reason for this, and we will continue to dive deeper and deeper into the concepts, the reasons for use, and examples of use.

Last I left off, we covered topics related to Custom Services. In this post, I want to take and cover concepts around the consumption of document services.

As I referenced in a previous blog post, there is a great write up already, that covers the technical steps needed to show off how to consume document services, specifically around the Product Data, or the EcoResProductService.

This write up, found here: Product-item data management services, focuses on the consumption of the EcoResProductService, among others, for creating product data, and even releasing that into a given legal entity. Since this is a very nice write up, I will not focus this post on repeating the same effort.

Instead, what I want to focus on, is consuming the EcoResProductService, from within X++, from within AX, for creating Product Master Data. So lets jump right into this.

First thing that we need is an instance of AX 2012, and a project. We will then need, within that project, a custom service group to deploy the out-of-the-box EcoResProductService via. After that, we will need a class, to contain the business logic, and finally for our example a job that kicks off that classes method, and therefore logic for creating our Product Master.



As you can see, from the above image, we have just that for our example. Now since there is nothing we need to really do to the EcoResProductService, lets move into the business logic for creating a Product Master record.

First we need to start with the variables that will be involved in, enabling this.:



You will notice, we have an EcoResProductService variable, which is used to perform the operations that are enabled for a given service, in this case document service, and for our need the create() function.

Next you will see we have a variable that presents the EcoResEcoResProduct object. This is what the EcoResProductService will accept in it's create call that we will see in a bit. This class extends from the AifDocument class, as you can see from the below image.



After that, we have EcoResEcoResProduct_Product_Master class, which is extends from the EcoResEcoResProduct_Product class, which further extends from the AfStronglyTypedDataContainer. You can see this as well, in the image below here.



From there we have variables that represent the EcoResEcoResProduct_Translation, EcoResEcoResProduct_Identifier & EcoResEcoResProduct_ProductDimGroup objects. All of which are used to help make up an EcoResEcoResProduct_Product_Master object.

Now all of that is a mouth full! The point is, we have, in order to work and create Product Master entries, via the Document Services concept, is six objects. These six objects in our example abstract the complexity of the new EcoResProduct data structure, and enable within AX for business logic to move away, from even working directly with the table objects within AX itself. This is very powerful, and very useful.

Now lets move forward, since we have an understanding of the objects that will enable us to create a Product Master record, and look into the code that enables this.



First part of the code, we see in the above image, our logic that initializes the service for use. Then we move towards creating a new instance of the EcoResEcoResProduct object. After that we are creating our Product Master object, and finally filling some of it's parameters, via the use of parameter methods, with values.



Next we are making use of the objects that make up our Product Master object, in the ProdMast variable, for creating the needed: Translation, Identifier and for our example the Product Dimension Group.

You will notice that the code to enable this is ProdMast.createTranslation().addNew(); In doing this, and for the other examples this is true as well, we are creating a new instance of the Translation object, and adding it to the AfStronglyTypeDataContainerList that is represented for the ProdMast variable.



Finally, we come to the bottom of the code block for this business logic, and see we are setting the Variant Configuration Technology. Then adding our ProdMast object to the Strongly Type List for the EcoResProd variable. Finally calling the create method of the service object, passing in our ProdMast variable.

On calling this logic, we get our Product Master record, as shown below.:


So with a total of 17 lines of code, and 6 objects, we are working with Document Services from within AX 2012 itself, and creating Product Master data, without having to work directly with the table objects themselves.

This can then lend itself to be re-used over and over again, and finds value in integration work, as well as data migration. You can also, now take and compare the differences between C# and X++ code for using these document services. They are similar, in theory, but different in syntax and possible approach of course.

That's all I wanted to cover for this series, and for the introduction to services in AX 2012 this just about wraps things up. I plan on doing a summary post, bringing together all the work I've done, all the work Microsoft has done, as well as other community contributors. Till next time!



"Visit the Dynamics AX Community Page today!"