Split records into Multiple csv files using a Threshold percentage

Hi Gurus,
I have a requirement to split the data into two csv file from a table using a threshold value(in Percentage) .
Assume that If my source select query of interface fetches 2000 records , I will provide a threshold value like 20%.
I need to generate a csv1 with 400 records(20% of 2000) and the rest of the records into another csv2.
For implementing this I am trying to use the following process.
1) Create a procedure with the select query to get the count of records.
Total Records count: select count(1) from source_table <Joins> <Lookups> <Conditions>;
2) Calculate the Record count to first CSV using the threshold_value.
CSV1_Count=Total records count /threshold_value
3) Create a view that fetches the CSV1_Count(400) records for CSV1 as follows.
Create view CSV1_view as select Col1,Col2,Col3 from source_table <Joins> <Lookups> <Conditions>
Where rownum<=CSV1_Count;
4) Generate CSV1 file using View 'CSV1_View'
5) Generate CSV2 File using the Interface with same select statement (with columns ) to generate a CSV.
select Col1,Col2,Col3 from source_table ST <Joins> <Lookups> <Conditions>
Left outer join (Select Col1 from CSV1_View ) CS on CS.Col1=ST.Col1 where CS.Col1 is null;
Which gives the Total records minus the CS1_View records.
The above process seems a bit complex and very simple . If any changes in my Interface I also need to change the procedure (counts the no:of records).
Please provide your comments and feedback about this and looking for your inputs for any new simple approach or fine tune the above approach.
Thanks,
Arjun

Arjun,
This are my thoughts and Lets do it in 3 Steps
Step 1.  ODI Procedure
Drop table Temp_20 ;
Create table Temp_20 as select * from table where rownum < ( SELECT TRUNC( COUNT(1) /5) FROM TABLE ) .
[ ** This way iam fetching approx 20% of the table data and loading into Temp table . 1/5 th is 20%  so i am dividing count by 5
I don't believe View will help you especially with RowNum as if you run the same query with rownum < N the rows order might differ . so Temp table is great ]
Step 2 .  Use OdiSqlUnload  with select columns  from temp_20
Step 3 . Use again OdiSqlUnload  with  select columns from table where  ( uk keys ) not in ( selecy uk_keys from temp_20)
[** this way you can pick the remaining 80% ** and the data will be not repeat itself across 20% and 80% , as might happen with view ]
what do you think ?

Similar Messages

  • Splitting of a CSV File with Multiple Records into Multiple XML File

    Dear All,
    <b> I am doing a Scenario of CSV to XML Files. I am using BPM for the same. My incoming CSV File has got multiple records. I want to break this Multiple records into Multiple XML Files having one record each.</b>
    Can someone suggest how can I break this rather Split this into Multiple XML Files.
    Is Multimapping absoltely necesaary for this. Can't we do this without Multimapping. Can we have some workaround in the FCC parameters that we use in the Integration Directory.
    Kindly reply ASAP. Thanks a lot to all in anticipation.
    Pls Help.
    Best Regards
    Chakra and Somnath

    Dear All,
    I am trying to do the Multimapping, and have 0....unbounded also. Someways it is not working.
    <b>
    Smitha please tell me one thing...Assigning the Recordsets per Message to 1, does it mean that it will write multiple XML Files as I want.</b>
    Also I am usinf Set to Read only. So once the File is read it becomes RA from A. Then will it write the other Records.
    I have to use a BPM because there are certain dependencies that are there for the entire Process Flow. I cannot do without a BPM.
    Awaiting a reply. Thanks a lot in anticipation.
    Best Regards
    Chakra and Somnath

  • How to get DocSet property values in a SharePoint library into a CSV file using Powershell

    Hi,
    How to get DocSet property values in a SharePoint library into a CSV file using Powershell?
    Any help would be greatly appreciated.
    Thank you.
    AA.

    Hi AOK,
    Would you please post your current script and the issue for more effcient support.
    In addition, to manage document set in sharepoint please refer to this script to start:
    ### Load SharePoint SnapIn
    2.if ((Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue) -eq $null)
    3.{
    4. Add-PSSnapin Microsoft.SharePoint.PowerShell
    5.}
    6.### Load SharePoint Object Model
    7.[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.SharePoint”)
    8.
    9.### Get web and list
    10.$web = Get-SPWeb http://myweb
    11.$list = $web.Lists["List with Document Sets"]
    12.
    13.### Get Document Set Content Type from list
    14.$cType = $list.ContentTypes["Document Set Content Type Name"]
    15.
    16.### Create Document Set Properties Hashtable
    17.[Hashtable]$docsetProperties = @{"DocumentSetDescription"="A Document Set"}
    18.$docsetProperties = @{"CustomColumn1"="Value 1"}
    19.$docsetProperties = @{"CustomColum2"="Value2"}
    20. ### Add all your Columns for your Document Set
    21.
    22.### Create new Document Set
    23.$newDocumentSet = [Microsoft.Office.DocumentManagement.DocumentSets.DocumentSet]::Create($list.RootFolder,"Document Set Title",$cType.Id,$docsetProperties)
    24.$web.Dispose()
    http://www.letssharepoint.com/2011/06/document-sets-und-powershell.html
    If there is anything else regarding this issue, please feel free to post back.
    Best Regards,
    Anna Wang
    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected]

  • Splitting TempDB into multiple data files.

    To avoid contention we have to split tempdb into multiple data files. But as for case suppose, there is 20 GB total space is on the drive containing 1 tempdb data file of 15 GB. And I have to create 3 more tempdb data files, and as recommendation all files
    should be of same size.Then how to handle this situation and configure all data files with same size?
    Pranshul Gupta

    But as for case suppose, there is 20 GB total space is on the drive containing 1 tempdb data file of 15 GB. And I have to create 3 more tempdb data files, and as recommendation all files should be of same size.Then how to handle this situation and configure
    all data files with same size?
    So your goal is to have 4 tempdb files, each 5GB?  Below is a sample script to accomplish the task within the 20GB space constraint.
    --reduce size of existing file to 5GB
    ALTER DATABASE tempdb
    MODIFY FILE (NAME='tempdev', Size=5GB);
    DBCC SHRINKFILE('tempdev',5120);
    --add 3 new 5GB files
    ALTER DATABASE tempdb
    ADD FILE (NAME='tempdev2', FILENAME='D:\SqlDataFiles\tempdb2.ndf', Size=5GB);
    ALTER DATABASE tempdb
    ADD FILE (NAME='tempdev3', FILENAME='D:\SqlDataFiles\tempdb3.ndf', Size=5GB);
    ALTER DATABASE tempdb
    ADD FILE (NAME='tempdev4', FILENAME='D:\SqlDataFiles\tempdb4.ndf', Size=5GB);
    Dan Guzman, SQL Server MVP, http://www.dbdelta.com

  • How do I split a Large CSV file into Multiple CSV's Using Powershell

    I am a novice at powershell but this looks to be the best tool to do this task. have a csv file that looks like this:
    Date,Policy,Application
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    Is it possible to split this CSV into multiple CSV's based on "Application".
    Lets say the output might look like:
    None.csv
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    10/13/2014,No,None
    AppBiz.csv
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    11/14/2013,Yes,AppBiz
    PeopleBiz.csv
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    07/04/2013,No,PeopleBiz
    Any help would be greatly appreciated

    I think this might be what you want:
    Import-Csv applications.csv |
    Group Application |
    foreach {
    $_.Group | Export-Csv "$($_.Name).csv" -NoTypeInformation
    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
    Very nice! 4x faster..
    I doubt the OP will get what you just did there..
    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________
    Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

  • Exporting query results into a csv file using arabic and hebrew chars

    Hi,
    iv'e encountered a problem, using plsql to export a query into a csv file.. the arabic turns into question mark. Do you have any idea why?

    Usually this indicates a mismatch between client and database character set.
    How do you export this query?
    What is your database version?
    What is your client OS?
    T.

  • Adding row into existing CSV file using C#

    How to add row to existing CSV file using .NET Code.the file should not be overwrite,it need add another row with data.how to implement this scenario.

    Hi BizQ,
    If you only just write some data to CSV file. Please follow A.Zaied and Magnus  's reply. In general,we use CSV file to import or export some data. Like following thread and a good article in codeproject
    Convert a CSV file to Excel using C#
    Writing a DataTable to a CSV file
    Best regards,
    Kristin
    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click
    HERE to participate the survey.

  • Download a UNIX based CGI page into a CSV file using Powershell

    Hello,
    I have been struggling to get a powershell script to work.
    I have a web-based page which ends in .cgi and I want to copy the text from that page to a CSV file.
    Normally the below script works for .html or .aspx types of pages but this time I have been getting the 'The remote server returned an error: (401) Unauthorized' page. I think the error is because the page is Unix based and for some reason I am not
    able to automate the authentication process.
    Below is the Powershell script I have been using. I have already created the securestring for my password and saved in a temp folder.
    $wc=new-object system.net.webclient
    $password = get-content 'C:\temp\Scripts\SecureString.txt' | convertto-securestring
    $wc.Credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist "Domain\username", $password
    $wc.downloadfile("https://website.domain.com/cgi-bin/text_status_csv.cgi", "C:\temp\Scripts\output.csv")
    Currently I am able to successfully copy the data from the page over if I use
    $wc.Credentials = Get-Credential
    And then type in the username and password manually. I could really use some help to get this automated.
    Thanks in advance for your help

    A little searching and you will find hundreds of examples.
    https://www.google.com/#newwindow=1&q=powershell+persist+credentials
    ¯\_(ツ)_/¯

  • How do I create a new mailbox file, not folder. My primary mailbox file is getting too large and I want to split it into multiple physical files.

    I have done this multiple times in the past (I have several files - not folders - that contain mail). However due to the fact that II was somewhat brain damaged several years ago I can no longer remember (or figure out) how to do it. For instance, I have a "MoreJunque" mailbox file that is (from the "Properties" dialog) at mailbox:///C:/Users/Daniel Mathews/AppData/Roaming/Thunderbird/Profiles/bxk6ngnt.default/Mail/pop.att.yahoo.com/MoreJunque. However, my inbox is in mailbox:///C:/Users/Daniel Mathews/AppData/Roaming/Thunderbird/Profiles/bxk6ngnt.default/Mail/pop.att.yahoo.com/Inbox. These are unique files that contain folders.

    right click on your account on the left and select new folder.

  • How to write the JTables Content into the CSV File.

    Hi Friends
    I managed to write the Database records into the CSV Files. Now i would like to add the JTables contend into the CSV Files.
    I just add the Code which Used to write the Database records into the CSV Files.
    void exportApi()throws Exception
              try
                   PrintWriter writing= new PrintWriter(new FileWriter("Report.csv"));
                   System.out.println("Connected");
                   stexport=conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
                   rsexport=stexport.executeQuery("Select * from IssuedBook ");
                   ResultSetMetaData md = rsexport.getMetaData();
                   int columns = md.getColumnCount();
                   String fieldNames[]={"No","Name","Author","Date","Id","Issued","Return"};
                   //write fields names
                   String rec = "";
                   for (int i=0; i < fieldNames.length; i++)
                        rec +='\"'+fieldNames[i]+'\"';
                        rec+=",";
                   if (rec.endsWith(",")) rec=rec.substring(0, (rec.length()-1));
                   writing.println(rec);
                   //write values from result set to file
                    rsexport.beforeFirst();
                   while(rsexport.next())
                        rec = "";
                         for (int i=1; i < (columns+1); i++)
                             try
                                    rec +="\""+rsexport.getString(i)+"\",";
                                    rec +="\""+rsexport.getInt(i)+"\",";
                             catch(SQLException sqle)
                                  // I would add this System.out.println("Exception in retrieval in for loop:\n"+sqle);
                         if (rec.endsWith(",")) rec=rec.substring(0,(rec.length()-1));
                        writing.println(rec);
                   writing.close();
         }With this Same code how to Write the JTable content into the CSV Files.
    Please tell me how to implement this.
    Thank you for your Service
    Jofin

    Hi Friends
    I just modified my code and tried according to your suggestion. But here it does not print the records inside CSV File. But when i use ResultSet it prints the Records inside the CSV. Now i want to Display only the JTable content.
    I am posting my code here. Please run this code and find the Report.csv file in your current Directory. and please help me to come out of this Problem.
    import javax.swing.*;
    import java.util.*;
    import java.io.*;
    import java.awt.*;
    import java.awt.event.*;
    import javax.swing.table.*;
    public class Exporting extends JDialog implements ActionListener
         private JRadioButton rby,rbn,rbr,rbnore,rbnorest;
         private ButtonGroup bg;
         private JPanel exportpanel;
         private JButton btnExpots;
         FileReader reading=null;
         FileWriter writing=null;
         JTable table;
         JScrollPane scroll;
         public Exporting()throws Exception
              setSize(550,450);
              setTitle("Export Results");
              this.setLocation(100,100);
              String Heading[]={"BOOK ID","NAME","AUTHOR","PRICE"};
              String records[][]={{"B0201","JAVA PROGRAMING","JAMES","1234.00"},
                               {"B0202","SERVLET PROGRAMING","GOSLIN","1425.00"},
                               {"B0203","PHP DEVELOPMENT","SUNITHA","123"},
                               {"B0204","PRIAM","SELVI","1354"},
                               {"B0205","JAVA PROGRAMING","JAMES","1234.00"},
                               {"B0206","SERVLET PROGRAMING","GOSLIN","1425.00"},
                               {"B0207","PHP DEVELOPMENT","SUNITHA","123"},
                               {"B0208","PRIAM","SELVI","1354"}};
              btnExpots= new JButton("Export");
              btnExpots.addActionListener(this);
              btnExpots.setBounds(140,200,60,25);
              table = new JTable();
              scroll=new JScrollPane(table);
              ((DefaultTableModel)table.getModel()).setDataVector(records,Heading);
              System.out.println(table.getModel());
              exportpanel= new JPanel();
              exportpanel.add(btnExpots,BorderLayout.SOUTH);
              exportpanel.add(scroll);
              getContentPane().add(exportpanel);
              setVisible(true);
          public void actionPerformed(ActionEvent ae)
              Object obj=ae.getSource();
              try {
              PrintWriter writing= new PrintWriter(new FileWriter("Report.csv"));
              if(obj==btnExpots)
                   for(int row=0;row<table.getRowCount();++row)
                             for(int col=0;col<table.getColumnCount();++col)
                                  Object ob=table.getValueAt(row,col);
                                  //exportApi(ob);
                                  System.out.println(ob);
                                  System.out.println("Connected");
                                  String fieldNames[]={"BOOK ID","NAME","AUTHOR","PRICE"};
                                  String rec = "";
                                  for (int i=0; i <fieldNames.length; i++)
                                       rec +='\"'+fieldNames[i]+'\"';
                                       rec+=",";
                                  if (rec.endsWith(",")) rec=rec.substring(0, (rec.length()-1));
                                  writing.println(rec);
                                  //write values from result set to file
                                   rec +="\""+ob+"\",";     
                                   if (rec.endsWith(",")) rec=rec.substring(0,(rec.length()-1));
                                   writing.println(rec);
                                   writing.close();
         catch(Exception ex)
              ex.printStackTrace();
         public static void main(String arg[]) throws Exception
              Exporting ex= new Exporting();
    }Could anyone Please modify my code and help me out.
    Thank you for your service
    Cheers
    Jofin

  • How to get Document Set property values in a SharePoint library in to a CSV file using Powershell

    Hi,
    How to get Document Set property values in a SharePoint library into a CSV file using Powershell?
    Any help would be greatly appreciated.
    Thank you.
    AA.

    Hi,
    According to your description, my understanding is that you want to you want to get document set property value in a SharePoint library and then export into a CSV file using PowerShell.
    I suggest you can get the document sets properties like the PowerShell Command below:
    [system.reflection.assembly]::loadwithpartialname("microsoft.sharepoint")
    $siteurl="http://sp2013sps/sites/test"
    $listname="Documents"
    $mysite=new-object microsoft.sharepoint.spsite($siteurl)
    $myweb=$mysite.openweb()
    $list=$myweb.lists[$listname]
    foreach($item in $list.items)
    if($item.contenttype.name -eq "Document Set")
    if($item.folder.itemcount -eq 0)
    write-host $item.title
    Then you can use Export-Csv PowerShell Command to export to a CSV file.
    More information:
    Powershell for document sets
    How to export data to CSV in PowerShell?
    Using the Export-Csv Cmdlet
    Thanks
    Best Regards
    TechNet Community Support
    Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact
    [email protected]

  • HT201066 How can I split a long recording audio file into multiple audio files?

    Hi,
    How can I split a long recording audio file into multiple audio files?
    With Windows I was using Nero software, which is not available for Mac. Is there any similar app for free for Mac?
    Thanks!

    Hello ingiorgio
    You can import the track into GarageBand and then split the tracks there. Once that it is done you can highlight the ruler section and then share the song to iTunes to get them separated out. 
    GarageBand - Split regions in the Tracks area
    http://help.apple.com/garageband/mac/10.0/#gbnd76fcce04
    GarageBand - Share songs to iTunes
    http://help.apple.com/garageband/mac/10.0/#gbndfb96a96f
    Regards,
    -Norm G.

  • How To Split Large Excel or CSV Files into Smaller Files

    Does anyone know how to split a large Excel or CSV file into multiple smaller files?  Or, is there an app that will work with Mac to do that?

    split [-a suffix_length] [-b byte_count[k|m]] [-l line_count] [-p pattern] [file [name]]
    is a native Terminal command. Read up more on https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/ man1/split.1.html
    I prefer to use gSplit which is gnu coreutils.
    You can install gnu coreutils using homebrew.

  • Split MP3 file into multiple MP3 files

    I've been through the help section and is till cant seem to figure out how to split a large (2 hour) mp3 file into multiple mp3 files.
    I have a 2 hour MP3 recording (Stereo not mono), and I have added markers (by simply pressing keyboard shortcut 'M' to add marker), but now I would like to know how to split that one mp3 files into multiple files from the markers which I have made...
    All my marker types are "Cue" when i pressed 'M' at the locations I want the track to end/start

    You didn't say which version of the program you're using, but in general, there is a way to do this by adding one additional cue at the very end of the file, then "merging" the cues (which converts the space between each cue into a "range"), and batch processing each range out to a separate file.  (Make sure there's a cue at the beginning of the file, or at least the first material you want in a new file.)
    Now that you've done all that work, I'm going to tell you that what you're doing is probably a bad idea.  Opening any mp3 in Audition and resaving back to mp3 means you've doubled the losses and distortion inherent with the mp3 compression.  Audition can't do lossless editing of mp3s.  The bottom line: are you really splitting the file, and does the result have to be mp3 format?  If saving to a (wave file, or any other lossless format, then what I'm saying doesn't matter.
    In any case, "mp3 Direct Cut" is a terrific freeware program that does lossless mp3 editing, and I'd highly recommend you try it.  It's saved my neck a number of times.  See http://www.mpesch3.de.

  • Script to merge multiple CSV files together with no duplicate records.

    I like a Script to merge multiple CSV files together with no duplicate records.
    None of the files have any headers and column A got a unique ID. What would be the best way to accomplish that?

    OK here is my answer :
    2 files in a directory with no headers.
    first column is the unique ID, second colomun you put whatever u want
    The headers are added when using the import-csv cmdlet
    first file contains :
    1;a
    2;SAMEID-FIRSTFILE
    3;c
    4;d
    5;e
    second file contains :
    6;a
    2;SAMEID-SECONDFILE
    7;c
    8;d
    9;e
    the second file contains the line : 2;b wich is the same in the first file
    the code :
    $i = 0
    Foreach($file in (get-childitem d:\yourpath)){
    if($i -eq 0){
    $ref = import-csv $file.fullname -Header id,value -Delimiter ";"
    }else{
    $temp = import-csv $file.fullname -Header id,value -Delimiter ";"
    foreach($line in $temp){
    if(!($ref.id.contains($line.id))){
    $objet = new-object Psobject
    Add-Member -InputObject $objet -MemberType NoteProperty -Name id -value $line.id
    Add-Member -InputObject $objet -MemberType NoteProperty -Name value -value $line.value
    $ref += $objet
    $i++
    $ref
    $ref should return:
    id                                                         
    value
    1                                                          
    a
    2                                                          
    SAMEID-FIRSTFILE
    3                                                          
    c
    4                                                          
    d
    5                                                          
    e
    6                                                          
    a
    7                                                          
    c
    8                                                          
    d
    9                                                          
    e
    (get-childitem d:\yourpath) -> yourpath containing the 2 csv file

Maybe you are looking for