Script to merge multiple CSV files together with no duplicate records.

I like a Script to merge multiple CSV files together with no duplicate records.
None of the files have any headers and column A got a unique ID. What would be the best way to accomplish that?

OK here is my answer :
2 files in a directory with no headers.
first column is the unique ID, second colomun you put whatever u want
The headers are added when using the import-csv cmdlet
first file contains :
1;a
2;SAMEID-FIRSTFILE
3;c
4;d
5;e
second file contains :
6;a
2;SAMEID-SECONDFILE
7;c
8;d
9;e
the second file contains the line : 2;b wich is the same in the first file
the code :
$i = 0
Foreach($file in (get-childitem d:\yourpath)){
if($i -eq 0){
$ref = import-csv $file.fullname -Header id,value -Delimiter ";"
}else{
$temp = import-csv $file.fullname -Header id,value -Delimiter ";"
foreach($line in $temp){
if(!($ref.id.contains($line.id))){
$objet = new-object Psobject
Add-Member -InputObject $objet -MemberType NoteProperty -Name id -value $line.id
Add-Member -InputObject $objet -MemberType NoteProperty -Name value -value $line.value
$ref += $objet
$i++
$ref
$ref should return:
id                                                         
value
1                                                          
a
2                                                          
SAMEID-FIRSTFILE
3                                                          
c
4                                                          
d
5                                                          
e
6                                                          
a
7                                                          
c
8                                                          
d
9                                                          
e
(get-childitem d:\yourpath) -> yourpath containing the 2 csv file

Similar Messages

  • Merge Multiple CSV Files Into One and save it into new file

    Below code run in specific directory. However I'm looking a code which will select the folder  & save it into new sheet.
    Can you help me with this? I've searched for a solution for a long time but could not find anything which worked.
    The following is the start of the macro code
    Sub OpenCSV()
    Dim i As Integer
    ' change this next line to reflect the actual directory
    Const strDir = "C:\Documents and Settings\shekar\"
    Dim ThisWB As Workbook
    Dim wb As Workbook
    Dim ws As Worksheet
    Dim strWS As String
    Set ThisWB = ActiveWorkbook
    Set fs = Application.FileSearch
    With fs
    .LookIn = strDir
    .Filename = "*.csv"
    If .Execute(SortBy:=msoSortByFileName, _
    SortOrder:=msoSortOrderAscending) > 0 Then
    For i = 1 To .FoundFiles.Count
    Set wb = Workbooks.Open(.FoundFiles(i))
    strWS = wb.Sheets(1).Name
    wb.Sheets(1).UsedRange.Copy (ThisWB.Worksheets(strWS).Range("A1"))
    wb.Close False
    Next i
    Else
    MsgBox "There were no files found."
    End If
    End With
    End Sub

    The below will do the.But be careful if any of csv file name is more than 31 then Excel will not allow it to name the sheet with filename.
    Sub OpenCSV()
    Dim i As Integer
    ' change this next line to reflect the actual directory
    Const strDir = "C:\Documents and Settings\shekar\"
    Dim ThisWB As Workbook
    Dim wb As Workbook
    Dim ws As Worksheet
    Dim strWS As String
    Dim sFileName As String
    Set ThisWB = ActiveWorkbook
    With Application.FileDialog(msoFileDialogOpen)
    .InitialFileName = strDir
    .Filters.Clear
    .Filters.Add "CSV Files", "*.csv"
    If .Show Then
    For i = 1 To .SelectedItems.Count
    Set wb = Workbooks.Open(.SelectedItems(i))
    sFileName = .SelectedItems(i)
    Set ws = ThisWB.Worksheets.Add(before:=ThisWB.Worksheets(1))
    ws.Name = ActiveSheet.Name
    wb.Sheets(1).UsedRange.Copy ws.Range("A1")
    wb.Close False
    Next i
    Else
    MsgBox "There were no files found."
    End If
    End With
    End Sub
    Please do not forget to click “Vote as Helpful” if any post helps you and
    Mark as Answer if it solves the issue.

  • How do I merge multiple Excel files with more than one tab in each file using PowerQuery?

    Hello
    I have 12 Excel (.xlsx) files and each file has three identically named and ordered tabs in them. 
    I know how to merge multiple Excel files in a folder using M (those guides are all over the web) but how do I merge multiple Excel files with multiple (yet identically named and ordered) tabs? Surely it is possible? I just don't know how to do it in M.
    Cheers
    James

    What Laurence says is correct, and probably the best thing to do when the sheets have differing structures. Here is an alternate approach that works well when the sheets all have the same structure.
    When you first open the Excel file from Power Query, you can see its structure in the navigator at the right-hand-side of the screen. If you select the root (which is the filename itself) and click Edit, you'll see all the tabs in the sheet as a single table.
    You can now do filtering based on the Name, Item and Kind values. When you've reduced the set of things down to the sheets you want, select the Data column and say "Remove Other Columns". If the sheets don't have any header rows, you can just click the expand
    icon in the header and you'll be done.
    Otherwise, if the sheets have headers or if some other kind of sheet-level transformation is required against each sheet before doing a merge, you'll have to write some M code manually. In the following example, each sheet has a header row consisting of
    two columns: Foo and Bar. So the only step I need to perform before merging is to promote the first row into a header. This is done via the Table.TransformColumns operation.
    let
        Source = Excel.Workbook(File.Contents("C:\Users\CurtH\Desktop\Test1.xlsx")),
        RemovedOtherColumns = Table.SelectColumns(Source,{"Data"}),
        PromotedHeaders = Table.TransformColumns(RemovedOtherColumns,{{"Data", each Table.PromoteHeaders(_)}}),
        ExpandData = Table.ExpandTableColumn(PromotedHeaders, "Data", {"Foo", "Bar"}, {"Data.Foo", "Data.Bar"})
    in
        ExpandData

  • How to export data to multiple csv files?

    Hey Scripting Guys,
    As stated by the name I'm a novice at scripting.  Typically I'm able to resolve most of my scripting challenges by reading through your site or scouring the internet.  This challenge I haven't been able to resolve.  Please help!!
    I'm running a script (posted below) to grab data and export it to a csv file.  My challenge is that I want to run the script daily via task manager and have it create a new csv file either daily or weekly.  I'm having trouble with the scripting
    creating a new csv file.  How do I resolve this?  
    It would be beneficial to append the date to a standard file name, ex. c:\exportedcsv7-11-2014.csv; the next day it would be c:\exportedcsv7-12-2014.csv; and so on.
    Thank you in advance to any assistance.
    Respectfully,
    ScriptingNovice
    Get-Datastore -Name "*DS*" | Sort $_.name | Get-View | Select -ExpandProperty Summary | `
    Select Name,
    @{N=”FreeSpaceGB”;E={[Math]::Round($_.FreeSpace/1GB,2)}},
    @{N=”CapacityGB”; E={[Math]::Round($_.Capacity/1GB,2)}},
    @{N=”UncommittedGB”; E={[Math]::Round($_.Uncommitted/1GB,2)}},
    @{N=”ProvisionedGB”;E={[Math]::Round(($_.Capacity – $_.FreeSpace + $_.Uncommitted)/1GB,2)}},
    @{N=”Over-Provisioned-DS”;E={([Math]::Round($_.Capacity/1GB,2)) – ([Math]::Round(($_.Capacity – $_.FreeSpace + $_.Uncommitted)/1GB,2))}}| `
    select Name,CapacityGB,FreespaceGB,ProvisionedGB,UncommittedGB,Over-Provisioned-DS | export-csv -notype c:\vmds.csv

    Thank you for the information and the tip.  I took your advice and did some research on Strings, something that I do have trouble grasping.  I'm familiar with variables already and feel comfortable using them.  I also need to study .Net which
    I totally don't understand. 
    Since I appreciate your advice and guidance I'd like to know if I'm on the correct track.  Please look at my breakdown to see if I'm explaining it correctly.
    $d=Get-Date
    Here a variable is being created using the Get-Date cmdlet, if we execute $d the date will appear
    $d.ToString('dd-MM-yyyy')
    This converts the date into the format dd-MM-yyyy.
    I took the information and tips you provided then came up with this after reading about strings.
    export-csv -notype "c:\folder\vmds_$($d.ToString('MM-dd-yyyy')).csv"
    The double quotes evaluate the variables.  The single quotes do not evaluate anything they just show what's inside of them (a.k.a. literal string). The $() evaluate the expression in ('MM-dd-yyyy') before writing it.
    You are correct I definitely need to strengthen the foundation of my basics.  Thank you for the direction and advice.

  • Is it possible to run a script each time a .CSV file is updated?

    I am having trouble with a script I am trying to write, which is made up of two parts:
    A .CSV file generated by PHP/MySQL whenever an RSS feed is updated. The contents of the .CSV are overwritten each time the PHP runs.
    A DataMerge script, written in JavaScript.
    What I would like to do is run the DataMerge script each time the .CSV file is updated. Is this possible, perhaps using an Event Listener within the PHP to trigger the DataMerge script? Or am I approaching this in the wrong way? I'm grateful for any thoughts you might have. Thanks, F.H.

    I would use Rorohiko's APID ToolKit: http://www.rorohiko.com/wordpress/indesign-downloads/active-page-item-developer/
    It has fileChanged event that is sent after the observed external file has changed.
    sorry guys for "labouring the point", but I'm working on very advanced project - and I need to know what my tool can or can't do
    and in this case - using APID will break point 2.3 of EULA, right ? interaction with external datasource - without user action ?
    robin
    www.adobescripts.co.uk

  • External Table which can handle appending multiple csv files dynamic

    I need an external table which can handle appending multiple csv files' values.
    But the problem I am having is : the number of csv files are not fixed.
    I can have between 2 to 6-7 files with the suffix as current_date. Lets say it will be like my_file1_aug_08_1.csv, my_file1_aug_08_2.csv, my_file1_aug_08_3.csv etc. and so on.
    I can do it by following as hardcoding if I know the number of files, but unfortunately the number is not fixed and need to something dynamically to inject with a wildcard search of file pattern.
    CREATE TABLE my_et_tbl
      my_field1 varchar2(4000),
      my_field2 varchar2(4000)
    ORGANIZATION EXTERNAL
      (  TYPE ORACLE_LOADER
         DEFAULT DIRECTORY my_et_dir
         ACCESS PARAMETERS
           ( RECORDS DELIMITED BY NEWLINE
            FIELDS TERMINATED BY ','
            MISSING FIELD VALUES ARE NULL  )
         LOCATION (UTL_DIR:'my_file2_5_aug_08.csv','my_file2_5_aug_08.csv')
    REJECT LIMIT UNLIMITED
    NOPARALLEL
    NOMONITORING;Please advice me with your ideas. thanks.
    Joshua..

    Well, you could do it dynamically by constructing location value:
    SQL> CREATE TABLE emp_load
      2      (
      3       employee_number      CHAR(5),
      4       employee_dob         CHAR(20),
      5       employee_last_name   CHAR(20),
      6       employee_first_name  CHAR(15),
      7       employee_middle_name CHAR(15),
      8       employee_hire_date   DATE
      9      )
    10    ORGANIZATION EXTERNAL
    11      (
    12       TYPE ORACLE_LOADER
    13       DEFAULT DIRECTORY tmp
    14       ACCESS PARAMETERS
    15         (
    16          RECORDS DELIMITED BY NEWLINE
    17          FIELDS (
    18                  employee_number      CHAR(2),
    19                  employee_dob         CHAR(20),
    20                  employee_last_name   CHAR(18),
    21                  employee_first_name  CHAR(11),
    22                  employee_middle_name CHAR(11),
    23                  employee_hire_date   CHAR(10) date_format DATE mask "mm/dd/yyyy"
    24                 )
    25         )
    26       LOCATION ('info*.dat')
    27      )
    28  /
    Table created.
    SQL> select * from emp_load;
    select * from emp_load
    ERROR at line 1:
    ORA-29913: error in executing ODCIEXTTABLEOPEN callout
    SQL> set serveroutput on
    SQL> declare
      2      v_exists      boolean;
      3      v_file_length number;
      4      v_blocksize   number;
      5      v_stmt        varchar2(1000) := 'alter table emp_load location(';
      6      i             number := 1;
      7  begin
      8      loop
      9        utl_file.fgetattr(
    10                          'TMP',
    11                          'info' || i || '.dat',
    12                          v_exists,
    13                          v_file_length,
    14                          v_blocksize
    15                         );
    16        exit when not v_exists;
    17        v_stmt := v_stmt || '''info' || i || '.dat'',';
    18        i := i + 1;
    19      end loop;
    20      v_stmt := rtrim(v_stmt,',') || ')';
    21      dbms_output.put_line(v_stmt);
    22      execute immediate v_stmt;
    23  end;
    24  /
    alter table emp_load location('info1.dat','info2.dat')
    PL/SQL procedure successfully completed.
    SQL> select * from emp_load;
    EMPLO EMPLOYEE_DOB         EMPLOYEE_LAST_NAME   EMPLOYEE_FIRST_ EMPLOYEE_MIDDLE
    EMPLOYEE_
    56    november, 15, 1980   baker                mary            alice     0
    01-SEP-04
    87    december, 20, 1970   roper                lisa            marie     0
    01-JAN-99
    SQL> SY.
    P.S. Keep in mind that changing location will affect all sessions referencing external table.

  • Merge multiple pdf files into single pdf

    How to merge multiple pdf files into single pdf file through any programmatic languages? Kindly share with me.
    Thanks

    Easy PDF
    http://www.macupdate.com/app/mac/38090/easy-pdf
    How To: Combine multiple PDFs in Preview
    http://macenstein.com/default/2011/12/how-to-combine-multiple-pdfs-in-preview/
    How to merge pdf files in OSX
    http://macintoshhowto.com/leopard/how-to-merge-pdf-files-with-preview-in-leopard .html
    How to Combine Multiple PDFs Using Preview
    http://www.macobserver.com/tmo/article/how_to_combine_multiple_pdfs_using_previe w/

  • Appending multiple *.csv files while retaining the original file name in the first column

    Hi guys it's been awhile.
    I'm trying to append multiple *.csv files while retaining the original file name in the first column, the actual data set is about 40 files.
    file a.csv contains:
    1, line one in a.csv
    2, line two in a.csv
    file b.csv contains:
    1, line one in b.csv
    2, line two in b.csv
    output.csv result is this:
    I would like this:
    a.csv, 1, line one in a.csv
    a.csv, 2, line two in a.csv
    b.csv, 1, line one in b.csv
    b.csv, 2, line two in b.csv
    Any suggestions to speed up my hobbling attempts would be aprieciated
    Thanks,
    -SS
    Solved!
    Go to Solution.

    What you could do is given in the attachment.
    Started with 2 files :
    a.csv
    copy of a.csv
    Both with data :
    1;1.123
    2;2.234
    3;3.345
    Output :
    a.csv;1;1.123
    a.csv;2;2.234
    a.csv;3;3.345
    Copy of a.csv;1;1.123
    Copy of a.csv;2;2.234
    Copy of a.csv;3;3.345
    If you have more questions, just shoot
    Kind regards,
    - Bjorn -
    Have fun using LabVIEW... and if you like my answer, please pay me back in Kudo's
    LabVIEW 5.1 - LabVIEW 2012
    Attachments:
    AppendingCSV.JPG ‏73 KB

  • Merge multiple PDF files

    How to merge multiple PDF files with its corresponding review comments and stamps, which has stored as FDF files in WebDAV server?

    Bring each FDF file into a corresponding PDF.
    Use Acrobat to merge the PDF files into a new, separate PDF.
    Be well...

  • Merging Multiple PDF Files in C# using Acrobat 7.0

    I'm trying to merge multiple pdf files into one pdf file. Does anyone have sample or psuedo code for this?

    Basicly, there has not much changed since version 5 so you can use also vers. 8 SDK / IAC section.
    The command you to use is "insert", where the first page has to be opened and the other pages will be inserted.
    If you google for PdDoc.insert you may get tons of examples and maybe also in C.
    If you use Acro-Javascript via IAC you may find an example direct in the AcroJs helpfile.
    HTH, Reinhard

  • Merging multiple pdf files not available

    I want to merge multiple PDF files into one PDF file. I used to be able to do it under file>create pdf, but ever since my computer was re-built, that option is no longer under the file tab. Was this something extra that you had to pay for, or do I have to do something to get this add in. Thank you in advance

    Yeah you need one of the acrobat family. To be specific, Acrobat Standard, Acrobat 9 Pro or Acrobat Extended
    They have 30 day trial on adobe.com.  But check first the system requirements so that it won't mess up your computer.

  • Can I link multiple project files together?

    I'd like to link multiple project files together so that they will play one after another. Can you do that in Captivate?

    Hey Leslie,
    Yes, you can create several projects in Captivate and publish them as Flash (SWF) format. There is an aggregator project in Captivate where in you can combine all these published projects (SWF) into a single output.
    Do you have any other requirement as well?
    Thanks,
    Anjaneai

  • Script to merge 2 Indesign files on alternating pages

    Hello
    I have created a data merge document for a book I am writing (Luckily I already had everything in Excel) . I created a template with the style headings and graphics frames positioned and sized exactly where I wanted them.My master page had repeating elements etc.
    Loved the process and it worked beautifully. It literally saved me a weeks worth of formatting and inserting graphics work. However the file basically created the Left hand page ONLY of a 2 page spread.
    My understanding from studying tutorials on data merge is that they can only create ONE page output altho I know you can change it to 2 page spreads afterward.
    The second data merge created a file what would ideally be the right hand page of every spread.
    This is a reference/encyclopedia type book with each 2 page spread a distinct entity devoted to a subject
    I need to join the 2 files together.
    The first file would be the left hand or EVEN page
    The second file would be the right hand or ODD page.
    The document is 350 pages.Each page is the same size,margins and subject matter but with different headings and graphics
    (Dragging pages would be too tedious and I have had problems with ID crashing. Don't want to go there again)
    I did read an answer by Dave Saunders (  9. Nov 1, 2008 5:03 PM (in response to Al Ferrari) Re: Script to interleave two InDesign files of equal page counts )
    Would that work do you think?
    I have ID CS4 6.0 running on windows vista.
    I have ZERO scripting experience but I can follow directions.
    Is there any way, scripting wise, I could (or should) accomplish this?
    Thank You in Advance for any help
    Lauren

    My understanding from studying tutorials on data merge is that they can only create ONE page output altho I know you can change it to 2 page spreads afterward.
    That isn't true. Data merge can be done with many pages. See my response to this post:
    http://forums.adobe.com/message/3909894#3909894
    In response to the OP's script request, i can't be of any help.
    What i'm hoping is that after reading how to make a data merge with both left and right hand pages from the post i've linked, that the data could be merged again... assuming there was little (or no) post-processing of the file after the merge was performed.

  • How would I merge two csv files in Powershell

    Hello,
    I have two CSV Files.  Each has a list of virtual machines in them. One CSV file has the name of the VM as well as details about the RAM, CPU, VM Details (from VMM).  The other CSV File has the name of the VM as well as the IP address.
    How would I merge these two together?  For any VM's that do not exist in either file I wouldn't want to drop these VM names either. 
    Thanks.

    Thank you mjolinor for your suggestion.  This is not quite working for me.  What I'm seeing in the csv3.csv file all the  column details from the $firstCSV, and I'm seeing the headings included from the $secondCSV but all the details
    of the headings from the $secondCSV are empty.  Here is some more information for you:
    My $firstCSV file has the following headings:
    VMName  ChargedWA  BillingWA  VMOperatingSystem  VMHost  vCPUs  MemoryGB  StorageGB 
    My $secondCSV file has the following headings:
    VMName  IPAddress  SubnetMask  Gateway  DNSServers  MACAddress  device0  Totalspce0  freespce0  device1  Totalspce1  freespce1  device2  Totalspce2  freespce2
    Each CSV input file has in common the column of VMName.  Most values in VMName are common to both input files but there are some values in VMName that are not in the other input file.  I'm looking to find the following:
    When the same VMname exists in both files, write out all columns for that record from both files into the merged file.
    When a VM name is in $firstCSV but not the second...still write out this record with the values from the $firstCSV.  The column headings from the $secondCSV would be empty for this record.
    When a VM name is in $secondCSV but not the first...still write out this record with the values from the $secondCSV.  The column headings from the $firstCSV would be empty for this record.
    It's funny how I think I've described my issue in my initial post but as I describe it more I further define the details.  I hope I've described my issue for all to understand.
    Thank you.

  • How to Merge multiple XML  files in one file ( Env: XML Publisher 5.6.2)

    All,
    I have recently started working on XML publisher and have developed 3 reports in last 2 days using XML Publisher and integrating them with Concurrent programs.
    This is a great tool.
    I have got another requirement, where i need to use xml file generated by multiple run of same report with various parameters and then merge all xml file to a single report. Developing the whole custom process will take very long time and sure will have bugs in it. Instead i was thinking to use xml file generated by Oracle report itself.
    Report "US Gross to net summary" generates xml output in standard output directory and then show output in PDF file. I have 7 such file generated for each payroll. I want to merge output of xml into a single xml so that i can create single report having data from all 7 xml files showing me All payroll output in a single report.
    Can someone please guide me , how can i read xml file data from the output directory of a seeded concurrent program and how to manipulate data in it.
    Thanks
    Ankur

    Hi Tim,
    Thanks for replying. I have looked for "PDFBookBinder class" in xml publisher user guide for ver 5.6.2. I didn't get any reference of this text. Can you please guide me to a tutorial/link where i can get more information about this class.
    Also, i originally thought of similar to your second logic, as my design basis. Oracle process generates the xml file in output directory which i can get. What i didn't get is how do i "pick them up and merge" using publisher. Also, is there way to do this merging process using pl/sql ? Can you please give little more information on your second approach.
    My original plan of action is that i will create a report set in which i will call oracle seeded report for all 7 payrolls in a sequential manner. Then using the child requests of the report set i will get to 7 xml files generated by seeded oracle process. Then the piece i am not sure of , i will use those 7 files to generate a single xml file having payroll name as tree top for each output. Once single xml is ready, i can easily design a template and register the process to generate output as Excel.This process will not require me to actually change any data or do any calculation. It will only reformatting the feilds we see and abiity to see all 7 payroll at one time rather then entering these numbers manually into an excel to do analysis.
    Please provide your feedback, if you think above plan is not feasible or need corrections.
    Best Regards,
    Ankur

Maybe you are looking for