How do we eliminate duplicate records in a flat file without using Sorter and Aggregator?

Answer Posted / isha

Select all source rows.
The Dynamic Lookup transformation builds the caches from the target table.
When the lookup evaluates a row from the source that does not exist in the lookup cache, it inserts the row into the cache and assigns the NewLookupRow output port the value of 1. When the lookup evaluates a row from the source that exists in the lookup cache, it does not insert the row into cache and assigns the NewLookupRow output port the value of 0.
The filter in this mapping checks if the row is a duplicate or not by evaluating the NewLookupRow output port from the Lookup. If the value of the port is 0, the row is filtered out, as it is a duplicate row. If the value of the port is not equal to 0, then the row is passed out to the target table.

Is This Answer Correct ?    1 Yes 0 No



Post New Answer       View All Answers


Please Help Members By Posting Answers For Below Questions

Describe expression transformation?

641


What is a surrogate key?

569


Why do we need informatica?

565


What is session task and command task?

659


what is the complex transformation you used in informatica

2205






How to convert multiple rows to single row (multiple columns) in informatica

907


IN SCD1, insource we have 10 billion records and in the first day its uploaded successfully and in the second day its taking time to upload because some records it might get update or insert new records. As a developer what will be the better solution for this??

571


Can we create a Integration service without a Repository service and vice versa?

734


how can we find the bottle neck in SQL Query in SOURCE QUALIFIER, and how can we tune it..?

5885


What is an aggregator transformation?

624


How can repository reports be accessed without sql or other transformations?

741


What are the mapping parameters and mapping variables?

631


what is the hint? how to use it to reduce the query processing in program?

6359


what is informatica metadata?

609


How to handle decimal places while importing a flatfile into informatica?

732