Friday, May 8, 2015

How to Append Multiple Files into Single File in Talend Studio !!

In this tutorial I will show you how to append multiple csv files of the same format into a single csv file.
Problem Statement :- Suppose you are getting same input format files for different countries on daily basis and now you want to consolidate or append all the countries file data into one input file. 

In below job I have taken three input csv files(with same schema) they are as follows:--

Student_List1

student_id;student_name;student_branchid
101;Sameer Chowdhary;1
102;Aditya Tiwari;1
103;Gaurav Tiwari;1
104;Shashi Singh;1
105;Yogesh Mishra;5
106;Ankit Gupta;8
107;Mohit Sharma;9
108;Rajesh Soni;8
109;Rohit Sinha;1

Student_List2

student_id;student_name;student_branchid
110;Radha Singh;2
111;Richa Swankar;9
112;Santosh Tiwari;4
113;Gaurav Tiwari;1
114;Mohammad Singh;1
115;Prachi Mishra;5
116;Duddu Gupta;3
117;Mahi Sharma;9
118;Renuka Soni;6
119;Swati Sinha;1

Student_List3

student_id;student_name;student_branchid
120;Ravi Mahlotra;7

1.So firstly create a new job by right clicking  on Job Designs in the Repository window and select “Create job”.
2.Create a schema of this three files by going to Metadata > FileDelimited > Create FileDelimited.
3.Drag and drop created metadata of all three files to the job design one by one then select tFileInputDelimited option from the pop window.
4.Drag and drop tFileOutputDelimited from the palette.
5.Connect each component as shown in the screenshot.


6. Double click on student_list1(tFileInputDelimited_1) component to open the Basic settings properties .The screen would look like below:--
And do same process for other two tFileInputDelimited.


7. Open the componnet properties of tFileOutputDelimited :--
8. Write the File Name by clicking .... button where you want to store your single csv file.
9. Tick append check box to append the data to existing file instead of creating new file every time.
Note: While running job for multiple files, All input files data will be appended to single file with this append check box.
10. Check include header so that single file will appear with header column.

And do same process for other tFileOutputDelimited but note that you provide same File Name because we have to append all three files into single file.


Now our job design is complete and we just need to run the job to load all three files into single csv file.

1 comment:

  1. Hi,
    Nice Article, i have requirement to append 25 files, all files are in , delimited format, do i have to create 25 delimited metadata in talend.

    thanks

    ReplyDelete