There are several ways of achieving this. We can do it through expression

How do we eliminate duplicate records in a flat file without using Sorter and Aggregator?

Question Posted / priyank

7 Answers
21956 Views
iNautix, I also Faced
E-Mail Answers

Answer Posted / priyank

There are several ways of achieving this. We can do it
through expression transformation and other is look up on
the target.

Expression transformation:

Create ports,

Var_PREV_KEY=Key
Var_CURR_KEY=Var_PREV_KEY
Var_CHK_DUPLICATE --> IIF(Var_CURR_KEY=Key,'DUP','NODUP')
OUT_DUPLICATE --> Var_CHK_DUPLICATE

Note: I have taken a scenario where the target table
contains only 1 Key. In case of multiple keys, will have to
create a few more Variable ports for both CURR and PREV and
in the Var_CHK_DUPLICATE port, we need to add those checks
with an 'AND' operator.E.g. For 2 keys,

Var_PREV_KEY1=Key1
Var_CURR_KEY1=Var_PREV_KEY1
Var_PREV_KEY2=Key2
Var_CURR_KEY2=Var_PREV_KEY2
Var_CHK_DUPLICATE --> IIF(Var_CURR_KEY1=Key1 AND
Var_CURR_KEY2=Key2,'DUP','NODUP')
OUT_DUPLICATE --> Var_CHK_DUPLICATE

If the Informatica version is Unix installation, then in
the pre session command you can give an unix command to
remove the duplicates from the file like

sort <file_name> | uniq > <file_name>.new

Hope it helps.

Is This Answer Correct ?

4 Yes

12 No

Post New Answer View All Answers

Please Help Members By Posting Answers For Below Questions

What is the status code in stored procedure transformation?

1231

Which is the t/r that builts only single cache memory?

1236

What is a repository manager?

1146

What are the differences between oltp and olap?

1015

What is data transformation manager process?

1220

What are active and passive transformations?

1128

What are the basic requirements to join two sources in a source qualifier transformation using default join?

1208

I am new to informatica and learning it,can anybody please tell me how we receive source as flat file in informatica,from where we get this flat file?

1815

Did u used latest transformations of 8.6.0? for what?

1924

difference between repository database and repository service?

1219

What are the steps involved in the migration from older version to newer version of Informatica Server?

2150

Which means the first record should come as last record and last record should come as first record and load into the target file?

1137

How can we use batches?

1062

Whats there in global repository

1736

Differentiate between reusable transformation and mapplet.

1141