AX 2012 – Data Migration Framework: Step-by-step setup
Recently I wrote about the release on InformationSource, of the Microsoft Dynamics AX 2012 Data Migration Framework. In that post, I mentioned that I would continue to cover this release, including walking through the process of setup, configuration and use. In this post I plan to start that coverage as promised, with a step-by-step setup of the Data Migration Framework.
Visit Hillstar Business Intelligence (www.HillstarBI.com) in order to truly unlock your data trapped in your Microsoft Dynamics investment. With our value driven business intelligence strategy Hillstar help you transform into a data informed company.
The idea with this post, is to walk you through the screens, points of interest, through the full compile and finally to the point of configuration. At that point, we will continue later with a new post on the configuration and use of the Data Migration Framework.
Now that we understand the purpose, and scope of this post, lets get started. First off, of course you need to download the Data Migration Framework from InformationSource. After having downloaded the Data Migration Framework, the next step is to extract the contents of the Zip file.
At this point you will want to launch the Setup for the Data Migration Framework.
Moving from here a decision must be made on what is to be installed.
Since I’m making the choice to install both Server and Client on a development instance of AX 2012 I will leave both checked. It’s worth to note the message contained within the first dialog box, around the fact that the server part must be installed on an AOS.
Moving along we must select the path in which the setup will install the application files to.
After this, you should reach the final screen, indicating that the setup program has completed it’s task.
When clicking on the finish button, the installation log file opens up for you. It’s worth scanning through to see if there are any errors.
Now that we have completed the install, we need to double check that we have the Data Migration Framework bits ready for our use. Do this by browsing to the location in which you had chosen previously. In doing so, you should see a folder with similar contents as shown below.
Now that we have verified that install completed successfully, lets move towards a point in the install documentation on TechNet.
Specifically there is a warning, that states: “Because the staging environment is highly normalized and may require significant processing bandwidth, we recommend that you increase the Maximum buffer size setting for your environment while you migrate data. Use the Server configuration utility to set the value.” Lets take the advice of Microsoft and do just that.
This means you need to launch the server configuration utility, go to the database tuning tab, and change the buffer size from 24, to say 48.
You will notice that in making such a change Microsoft warns of doing so. Since we are doing this on purpose, we can ignore this warning. However it’s valid to point out that after your done with the data migration framework, it’s best to change this setting back to 24. Further, you might find that 48 is still not big enough and you will need to increase further.
After doing this, and clicking apply you should take the option of allowing the configuration utility to restart the Microsoft Dynamics AX 2012 AOS instance.
Now we have to move to importing the model file that came as part of the install bits for this offering. We perform this install either by using the PowerShell command console, or the AXUtil application. I’ve made the choice of using the AXUtil application.
Finally after a successful install of the model file, to your target instance of Microsoft Dynamics AX 2012, you should launch AX. In doing so, it will detect a new model as been installed, and ask how you wish proceed. For this, we want to make the choice of ‘Compile and synchronize’ and then click ok.
After the successful completion of the full compile and synchronization of the database, next action to take is a full CIL compile. This should take roughly 15 +/- minutes to complete. After completing this step, your instance of AX 2012 has the newly released Data Migration Framework deployed and setup. You should see the something similar to the following when done.
Now that we have the Data Migration Framework deployed and setup, the next steps goes towards configuration. This is where creating entity maps, and understanding source through to target needs great focus, clarity and I can stress enough human workflow.
Well that’s all for this post. I hope you find it useful and helpful in speeding up your adoption and use of the Microsoft Dynamics AX Data Migration Framework. Keep in mind that this is still in beta, and there will be issues. So keep a watch on InformationSource and further this blog for updates. Till Next Time!
Visit Hillstar Business Intelligence (www.HillstarBI.com) in order to truly unlock your data trapped in your Microsoft Dynamics investment. With our value driven business intelligence strategy Hillstar help you transform into a data informed company.
Labels: AX 2012, AX 2012 Data Migration Framework, Data Migration, Dynamics AX, Dynamics AX 2012, Entities, InformationSource, Master Data, MDM, Microsoft, Staging Tables
39 Comments:
I've installed the model and i see all the menus in Ax. After the setup i try to import a file from demo folder. During the import in the staging table i have this error:
Error occured while adding control flow components to package. This could be due to incorrect metadata.
You have some idea? It' possible a my mistake?
Thanks
Hi Eddy
This error would come if there is mapping issue in source to staging step.
Can you please go to Processing groups -> Entities -> Modify source mapping. Go to mapping details, if the grid is empty or there is a row which does not have both source and staging fields defined , then that need to be mapped correctly.
To auto generate mapping , you would have to - Processing groups -> Entities -> Generate source Mapping. After that go to modify source mapping to make sure mapping is correct.
Regards
Mudit
Hi Mudit,
i try to import the unitofMeasure delimited Demo file(the most simple file to import) , i generate the source mapping and after i check on modify sourcemapping if there are wrong rows. (not found wrong rows).
When i click to obtain a preview
(Preview source file) i have always the error that i posted earlier.
There is other setup to do?
(my environment have CU3 installed it'a problem?)
Thanks
Hi Eddy
Do you see any details in 'Preview error details' , this button is next to preview.
Can you put ur file as text over blog with just tags and one row of data?
Regards
Mudit
Hi Mudit,
Preview error details form is empty, i use the file "UnitOfMeasureEntity_Basic"
that is present in the installation path. ("C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (Beta)\DemoFiles\Delimited")
Thanks , i really appreciate the time you spent for my problem.
Hi Mudit,
Preview error details form is empty, i use the file "UnitOfMeasureEntity_Basic" that is present
in the installation path. ("C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (Beta)\DemoFiles\Delimited")
Thanks , i really appreciate the time you spent for my problem.
Hi Mudit,
Preview error details form is empty, i use the file "UnitOfMeasureEntity_Basic" that is present
in the installation path. ("C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (Beta)\DemoFiles\Delimited")
Thanks , i really appreciate the time you spent for my problem.
I just tried on my local , unable to repro this.
can you please confirm what you see -Processing groups -> Entities -> Modify source mapping. Go to mapping details
Source field ->Staging field
DecimalPrecision ->DecimalPrecision
IsBaseUnit ->IsBaseUnit
IsSystemUnit ->IsSystemUnit
Symbol ->Symbol
SystemOfUnits ->SystemOfUnits
UnitOfMeasureClass ->UnitOfMeasureClass
UOMInternalCode_CodeSymbol ->UOMInternalCode_CodeSymbol
UOMTranslation_Description ->UOMTranslation_Description
I'm also having the problem Eddy has, and I also have CU3 installed. The problem is exactly the same: I get an error when I click on Preview Source File (Error occured while adding control flow components to package. This could be due to incorrect metadata.) and when I click on Preview error details, the grid is empty (no errors shown).
hi,
the mapping is correct...
i'll try to install in another environment.
CIAO
I think I have the same problem as Eddy. I also tried to import a file from demo folder and there was an exception. The mapping seems to be correct. Each row has both source and staging fields defined. What can be wrong? Thanks a lot for helping.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Exception: DMF002 ---> Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException: Das Flat File Source-Element ist in der PipelineComponentInfos-Auflistung nicht vorhanden.
---> System.Runtime.InteropServices.COMException: Das Flat File Source-Element ist in der PipelineComponentInfos-Auflistung nicht vorhanden.
at Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSPipelineComponentInfos100.get_Item(Object Index)
at Microsoft.SqlServer.Dts.Runtime.PipelineComponentInfos.get_Item(Object index)
--- End of inner exception stack trace ---
at Microsoft.SqlServer.Dts.Runtime.PipelineComponentInfos.get_Item(Object index)
at Microsoft.Dynamics.AX.DMF.SSISHelper.DMFPackage.AddFlatFileSource(MainPipe dataFlowTask, String name)
at Microsoft.Dynamics.AX.DMF.SSISHelper.DMFPackage.AddControlFlowComponent(String backupTable, List`1 packageColumns, String entityname, String definitiongroup, String executionid, List`1 lstFlatFileErrorColumns, List`1 lstOLEDBErrorColumns, String errorTableName)
--- End of inner exception stack trace ---
at Microsoft.Dynamics.AX.DMF.SSISHelper.DMFEntity.MigrateEntity()
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(IRuntimeMethodInfo method, Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeType typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Dynamics.AX.ManagedInterop.ClrBridgeImpl.InvokeClrInstanceMethod(ClrBridgeImpl* , ObjectWrapper* objectWrapper, Char* pszMethodName, Int32 argsLength, ObjectWrapper** arguments, Boolean* argsAreByRef, Boolean* isException)
TYPE:
System.Reflection.TargetInvocationException
Ok can you please confirm "Integration Services"(SSIS) for SQL is installed on AOS box.. I think we miss this in help and error is misleading..
Regards
Mudit
We ran into the same issue, SSIS was installed on the server that hosts the database. Installing SSIS on the server that hosts the AOS resolved the error.
Thx a lot!
Now works Great!
THANX
Thanks we will get the documentation updated.
Regards
Mudit
Hi all, I am facing few issues related with DMF data importing.
1. DMF is not allowing me to import same main account id's in multiple companies. DMF is updating the same record instead of creating a new record. I have provided different Ledger chart of accounts for the different companies.
2. Target mapping is not available for the Out-Of-Box DMF CustomerAddress entity.
Goto -> Target entities -> select Customer address. Click on Modify target mapping menu button. Mapping from staging to taget is missing.
Any thoughts?
Thanks in advance,
Sreejith V Ramachandran, Bangalore.
Hi
1. For Main account -
Will look into this one.
2.
You should see GeneratePostalAddress block in middle. If that is there , it should work. Please try with demo file.
REgards
Mudit
Hi All,
I was able to import the demo Customer data a couple of days ago. Now I am also getting the same error that Eddy was getting, "Error occured while adding control flow components to package. This could be due to incorrect metadata". I do have SSIS installed on the AOS but any Processing Group that I try, UOM, Customers, etc., I get this error. Are there any other prerequisites that need to be installed? SSIS is installed on the AOS but no other SQL Server components.
Thanks!
Tim
hi,
we installed it on our AX2012 environment, but we are using SQL Server 2012.
When performing the preview or import into staging tables the following error comes out:
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SqlServer.ManagedDTS, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SqlServer.ManagedDTS, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91'
at Microsoft.Dynamics.AX.DMF.SSISHelper.DMFEntity.ShowPreview()
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
I checked and can confirm that the required assembly is not present.
Is there some workaround the tool can be used with SQL Server 2012?
Thanks in advance
@Tim - the error could also come if mapping is not correct. Please verify mapping is correct.My earlier posts have details for those.
@thomas - support for SQL 2012 is not there in Beta version, we plan to have it in future version. Some people worked arround this by installing required SQL 2008 components on AOS box. i have not tried that.
Regards
Mudit
@Mudit;
Whats the purpose of the mandatory checkbox @ entity attributes (XML). I've set its value to 'yes' and imported into AX where data was missing in this column for certain records. No warning appeared. Is this how this option is meant? Is there a way to force some fields to be filled which database logic doesn’t requires that from data architecture perspective but if needed for business purposes?
@Jeorme
The mandatory attribute in beta is just to highlight that it is recommended to have value in that column in file, however that is not enforced yet.
"Is there a way to force some fields to be filled which database logic doesn’t requires that from data architecture perspective but if needed for business purposes?"
- do you want this in source to staging? or you still want data to be copied to staging and then in staging when you press validate you should get error? Because feedback we got was to implement validations in staging as it is convinient to fix data there.. We also thought of bulk updates on staging data like update a column on all records with some value. That kind of support we have added with conversion values in source to staging.
Regards
Mudit
The Beta won't install on a Windows 2008 R2 Server with Terminal services installed. The AX client installation is on a D:\ Drive. Installation of the client and server components are both enabled but not editable and the next button is disabled.
is this a known issue?
This comment has been removed by the author.
For Installation issue - Windows 2008 R2 Server with Terminal services
Can you please make sure you right click and run as administrator, not aware about this issue, but would investigate.
Workarround for now -
http://technet.microsoft.com/EN-US/library/jj225595
The following table describes how to proceed if the DLLs are not present for each installation type.
Installation type
How to proceed if DLLs are not present
Client
Copy the DLLs from the installation location (C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (Beta)) to the folder indicated.
Server
Copy the Microsoft.Dynamics.AX.DMF.SSISHelper.dll from the installation location (C:\Program Files\Microsoft Dynamics AX 2012 Data Migration Framework (Beta)) to the folder indicated.
Use the Assembly Registration Tool (Regasm.exe) to register the DLL. At a command line, enter:
regasm Microsoft.Dynamics.AX.DMF.SSISHelper.dll /tlb: Microsoft.Dynamics.AX.DMF.SSISHelper.tlb
Regards
Mudit
@Mudit
"No rows have been migrated to staging"
Hi,
this page seems to be a good source for help:)
I installed the DMF successfully and played around with the demo files. Everything works fine.
Now I want to use it to import data into custom tables. I created the Staging Table with the wizard "Create a custom entity for migration", created a demo text file etc. etc. (I followed the same logic as provided with the sample files).
When clicking Preview Source file I receive the error "No rows have been migrated to staging"
The error occurs in Class DMFStagingWriter Function ShowPreview
Line: "dataTable = dmfEntity.ShowPreview();"
thx for your help!
@Mudit
"No rows have been migrated to staging"
Hi,
this forum seems to be a good source for help:)
I installed the DMF successfully and played around with the demo files. Everything works fine.
Now I want to use it to import data into custom tables. I created the Staging Table with the wizard "Create a custom entity for migration", created a demo text file etc. etc. (I followed the same logic as provided with the sample files).
When clicking Preview Source file I receive the error "No rows have been migrated to staging"
The error occurs in Class DMFStagingWriter Function ShowPreview
Line: "dataTable = dmfEntity.ShowPreview();"
thx for your help!
Hi,
We've used the framework to create a custom entity, but when we move the data from staging to target, the EntityWriter finds the first record in the target table and updates it for each record in the source table.
Similar problem logged above by Sreejith Nair.
Any suggestions? Wrong indexes? Wrong table group?
Thanks a lot,
JFB
We have found a workaround for the exception "No rows have been migrated to staging".
It appears that notepad Unicode format is accepted by the data migration framework as Unicode, but then throws the exception. Changing the source file format to ANSI and unchecking the Unicode option on the source data format allows us to preview the file correctly.
awesome! Changing the format to ANSI and unchecking Unicode worked. Thank you so much JFB!
(hope somebody can report this to MS)
I have managed to import projects fine, and have managed to create a custom entity for contract and had it work fine. However I have created a custom entity for funding sources and when trying to get staging data I get the following error: "Error occured while adding control flow components to package. This could be due to incorrect metadata." I have checked the mapping and everything is mapped except the two used for the import (DefinitionGroup and ExecutionID). Any other thoughts on what would be causing this error?
Thanks,
Tracy
I have managed to import projects fine, and have managed to create a custom entity for contract and had it work fine. However I have created a custom entity for funding sources and when trying to get staging data I get the following error: "Error occured while adding control flow components to package. This could be due to incorrect metadata." I have checked the mapping and everything is mapped except the two used for the import (DefinitionGroup and ExecutionID). Any other thoughts on what would be causing this error?
Thanks,
Tracy
New version for Data Migration Framework has been released.
http://blogs.msdn.com/b/axsa/archive/2012/10/22/microsoft-dynamics-ax-2012-data-migration-framework-beta-2-0-is-now-available-on-information-source.aspx
Besides the already rich feature set, some of the key new features which we have added in this release are below -
1. ODBC as Source – with this it is possible to import data using ODBC.
2. AX as Source – It supports export/import , copy company for Data migration Framework entities.
3. New entity types – Direct Table and Composite Entity.
4. Parallel execution support from Staging to Target.
5. Service Interface – DMF can be executed by external applications
6. Error handling enhancements – skipping error rows, and detailed logs.
7. Number sequence support.
8. Removed requirement to install SSIS on AOS .
I just installed latest version of DMF Beta2.
1) Run into same issues described above with files neeeding to be saved as ANSI.
2) Running into following issue when copying from staging to target. DMF only copies one record. Same issue with data templates and my own template.
-1073450952SSIS.PipelineSSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on component "Flat File Source" (1) returned error code 0x80040E21. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages posted before this with more information about the failure.
Sorry I just posted the previoius post and the error was actually when you try to Preview Source File. I created the Source File bu selecting columns(custom template).
2) I could not figured it out so I started using the Vendor template and just updating the data.
3) File gets loaded and stored in staging table. The issue I have now is that NO RECORDS get copie to the target table(Vendor). Again I am using the std template. :(
Hi,
I'm using the beta 2 version to import customers and addresses. It looks that logistics section (mode of delivery, terms etc. that come for ShipCarrierAddress) are not fully implemented. They exist on the staging table but not on the target. Could someone explain how to add them? also how to add a custom table to the existing customer address entity. This one is based on view unlike most others.
Thanks
I'm trying to install DMF. I managed to successfully import the AX model. However, when I open AX client and choose Compile and Synchronize the process takes more tan 2 hours. Is this mormal? I am running AX2012R2A demo VM released in Decmber 2012 with 12GB RAM.
In the AXUtil command there should be no space between /file: and the "C:\
If a space is used an error message is displayed.
Hi, has anyone installed Beta 2 successfully and compiled it successfully?
I am using AX 2012 R2, SQL 2012 SP1 BI, Windows 2008R2.
(This is borrowed from another blog)
This is the exact error I get:
I am trying to install DMF beta 2 and successfully used the /conflict:Push option with axutil to import the axmodel file.
Everything completed successfully in the upgrade checklist until attempting the Compile into .NET Framework CIL which failed with 6 errors. The Compiler output error description was the same for each error being: "CIL generation: Object reference not set to an instance of an object"
Path = \XppIL
Method/Property name = XppIL
Diagnositc ID = Err:351
Infolog returned the error message "The CIL generator found errors and could not save the new assembly"
I can definitely confirm the application is installed. In fact, the axmodel imports fine (at least without error messages).
A site that I used for the the DMF download link and install guidance has this as a resolution:
This is an error regarding to Public Sector. The DMF axmodel overwrites the macro "DataContainerTypes". Just compare the macro with fpk and fpp layer and take the following code into var or usr layer:
//
#AxdCustomerTransactionDCT
#AxdAdvancedLedgerEntryDCT
//
after bringing the macro to upper layer, right click on each object thatthe error is showing and compile them individually, and also delete the CIL failed log files present in C:\Program Files\Microsoft Dynamics AX\60\Server\MicrosoftDynamicsAX1\bin\XppIL following path
Just compile and generate Full CIL - it should work.
Post a Comment
<< Home