Split records into Multiple csv files using a Threshold percentage
Hi Gurus,
I have a requirement to split the data into two csv file from a table using a threshold value(in Percentage) .
Assume that If my source select query of interface fetches 2000 records , I will provide a threshold value like 20%.
I need to generate a csv1 with 400 records(20% of 2000) and the rest of the records into another csv2.
For implementing this I am trying to use the following process.
1) Create a procedure with the select query to get the count of records.
Total Records count: select count(1) from source_table <Joins> <Lookups> <Conditions>;
2) Calculate the Record count to first CSV using the threshold_value.
CSV1_Count=Total records count /threshold_value
3) Create a view that fetches the CSV1_Count(400) records for CSV1 as follows.
Create view CSV1_view as select Col1,Col2,Col3 from source_table <Joins> <Lookups> <Conditions>
Where rownum<=CSV1_Count;
4) Generate CSV1 file using View 'CSV1_View'
5) Generate CSV2 File using the Interface with same select statement (with columns ) to generate a CSV.
select Col1,Col2,Col3 from source_table ST <Joins> <Lookups> <Conditions>
Left outer join (Select Col1 from CSV1_View ) CS on CS.Col1=ST.Col1 where CS.Col1 is null;
Which gives the Total records minus the CS1_View records.
The above process seems a bit complex and very simple . If any changes in my Interface I also need to change the procedure (counts the no:of records).
Please provide your comments and feedback about this and looking for your inputs for any new simple approach or fine tune the above approach.
Thanks,
Arjun
Arjun,
This are my thoughts and Lets do it in 3 Steps
Step 1. ODI Procedure
Drop table Temp_20 ;
Create table Temp_20 as select * from table where rownum < ( SELECT TRUNC( COUNT(1) /5) FROM TABLE ) .
[ ** This way iam fetching approx 20% of the table data and loading into Temp table . 1/5 th is 20% so i am dividing count by 5
I don't believe View will help you especially with RowNum as if you run the same query with rownum < N the rows order might differ . so Temp table is great ]
Step 2 . Use OdiSqlUnload with select columns from temp_20
Step 3 . Use again OdiSqlUnload with select columns from table where ( uk keys ) not in ( selecy uk_keys from temp_20)
[** this way you can pick the remaining 80% ** and the data will be not repeat itself across 20% and 80% , as might happen with view ]
what do you think ?
Similar Messages
-
Splitting of a CSV File with Multiple Records into Multiple XML File
Dear All,
<b> I am doing a Scenario of CSV to XML Files. I am using BPM for the same. My incoming CSV File has got multiple records. I want to break this Multiple records into Multiple XML Files having one record each.</b>
Can someone suggest how can I break this rather Split this into Multiple XML Files.
Is Multimapping absoltely necesaary for this. Can't we do this without Multimapping. Can we have some workaround in the FCC parameters that we use in the Integration Directory.
Kindly reply ASAP. Thanks a lot to all in anticipation.
Pls Help.
Best Regards
Chakra and SomnathDear All,
I am trying to do the Multimapping, and have 0....unbounded also. Someways it is not working.
<b>
Smitha please tell me one thing...Assigning the Recordsets per Message to 1, does it mean that it will write multiple XML Files as I want.</b>
Also I am usinf Set to Read only. So once the File is read it becomes RA from A. Then will it write the other Records.
I have to use a BPM because there are certain dependencies that are there for the entire Process Flow. I cannot do without a BPM.
Awaiting a reply. Thanks a lot in anticipation.
Best Regards
Chakra and Somnath -
How to get DocSet property values in a SharePoint library into a CSV file using Powershell
Hi,
How to get DocSet property values in a SharePoint library into a CSV file using Powershell?
Any help would be greatly appreciated.
Thank you.
AA.Hi AOK,
Would you please post your current script and the issue for more effcient support.
In addition, to manage document set in sharepoint please refer to this script to start:
### Load SharePoint SnapIn
2.if ((Get-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue) -eq $null)
3.{
4. Add-PSSnapin Microsoft.SharePoint.PowerShell
5.}
6.### Load SharePoint Object Model
7.[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.SharePoint”)
8.
9.### Get web and list
10.$web = Get-SPWeb http://myweb
11.$list = $web.Lists["List with Document Sets"]
12.
13.### Get Document Set Content Type from list
14.$cType = $list.ContentTypes["Document Set Content Type Name"]
15.
16.### Create Document Set Properties Hashtable
17.[Hashtable]$docsetProperties = @{"DocumentSetDescription"="A Document Set"}
18.$docsetProperties = @{"CustomColumn1"="Value 1"}
19.$docsetProperties = @{"CustomColum2"="Value2"}
20. ### Add all your Columns for your Document Set
21.
22.### Create new Document Set
23.$newDocumentSet = [Microsoft.Office.DocumentManagement.DocumentSets.DocumentSet]::Create($list.RootFolder,"Document Set Title",$cType.Id,$docsetProperties)
24.$web.Dispose()
http://www.letssharepoint.com/2011/06/document-sets-und-powershell.html
If there is anything else regarding this issue, please feel free to post back.
Best Regards,
Anna Wang
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Support, contact [email protected] -
Splitting TempDB into multiple data files.
To avoid contention we have to split tempdb into multiple data files. But as for case suppose, there is 20 GB total space is on the drive containing 1 tempdb data file of 15 GB. And I have to create 3 more tempdb data files, and as recommendation all files
should be of same size.Then how to handle this situation and configure all data files with same size?
Pranshul GuptaBut as for case suppose, there is 20 GB total space is on the drive containing 1 tempdb data file of 15 GB. And I have to create 3 more tempdb data files, and as recommendation all files should be of same size.Then how to handle this situation and configure
all data files with same size?
So your goal is to have 4 tempdb files, each 5GB? Below is a sample script to accomplish the task within the 20GB space constraint.
--reduce size of existing file to 5GB
ALTER DATABASE tempdb
MODIFY FILE (NAME='tempdev', Size=5GB);
DBCC SHRINKFILE('tempdev',5120);
--add 3 new 5GB files
ALTER DATABASE tempdb
ADD FILE (NAME='tempdev2', FILENAME='D:\SqlDataFiles\tempdb2.ndf', Size=5GB);
ALTER DATABASE tempdb
ADD FILE (NAME='tempdev3', FILENAME='D:\SqlDataFiles\tempdb3.ndf', Size=5GB);
ALTER DATABASE tempdb
ADD FILE (NAME='tempdev4', FILENAME='D:\SqlDataFiles\tempdb4.ndf', Size=5GB);
Dan Guzman, SQL Server MVP, http://www.dbdelta.com -
How do I split a Large CSV file into Multiple CSV's Using Powershell
I am a novice at powershell but this looks to be the best tool to do this task. have a csv file that looks like this:
Date,Policy,Application
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
Is it possible to split this CSV into multiple CSV's based on "Application".
Lets say the output might look like:
None.csv
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
10/13/2014,No,None
AppBiz.csv
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
11/14/2013,Yes,AppBiz
PeopleBiz.csv
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
07/04/2013,No,PeopleBiz
Any help would be greatly appreciatedI think this might be what you want:
Import-Csv applications.csv |
Group Application |
foreach {
$_.Group | Export-Csv "$($_.Name).csv" -NoTypeInformation
[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "
Very nice! 4x faster..
I doubt the OP will get what you just did there..
Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________
Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx -
Exporting query results into a csv file using arabic and hebrew chars
Hi,
iv'e encountered a problem, using plsql to export a query into a csv file.. the arabic turns into question mark. Do you have any idea why?Usually this indicates a mismatch between client and database character set.
How do you export this query?
What is your database version?
What is your client OS?
T. -
Adding row into existing CSV file using C#
How to add row to existing CSV file using .NET Code.the file should not be overwrite,it need add another row with data.how to implement this scenario.
Hi BizQ,
If you only just write some data to CSV file. Please follow A.Zaied and Magnus 's reply. In general,we use CSV file to import or export some data. Like following thread and a good article in codeproject
Convert a CSV file to Excel using C#
Writing a DataTable to a CSV file
Best regards,
Kristin
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey. -
Download a UNIX based CGI page into a CSV file using Powershell
Hello,
I have been struggling to get a powershell script to work.
I have a web-based page which ends in .cgi and I want to copy the text from that page to a CSV file.
Normally the below script works for .html or .aspx types of pages but this time I have been getting the 'The remote server returned an error: (401) Unauthorized' page. I think the error is because the page is Unix based and for some reason I am not
able to automate the authentication process.
Below is the Powershell script I have been using. I have already created the securestring for my password and saved in a temp folder.
$wc=new-object system.net.webclient
$password = get-content 'C:\temp\Scripts\SecureString.txt' | convertto-securestring
$wc.Credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist "Domain\username", $password
$wc.downloadfile("https://website.domain.com/cgi-bin/text_status_csv.cgi", "C:\temp\Scripts\output.csv")
Currently I am able to successfully copy the data from the page over if I use
$wc.Credentials = Get-Credential
And then type in the username and password manually. I could really use some help to get this automated.
Thanks in advance for your helpA little searching and you will find hundreds of examples.
https://www.google.com/#newwindow=1&q=powershell+persist+credentials
¯\_(ツ)_/¯ -
I have done this multiple times in the past (I have several files - not folders - that contain mail). However due to the fact that II was somewhat brain damaged several years ago I can no longer remember (or figure out) how to do it. For instance, I have a "MoreJunque" mailbox file that is (from the "Properties" dialog) at mailbox:///C:/Users/Daniel Mathews/AppData/Roaming/Thunderbird/Profiles/bxk6ngnt.default/Mail/pop.att.yahoo.com/MoreJunque. However, my inbox is in mailbox:///C:/Users/Daniel Mathews/AppData/Roaming/Thunderbird/Profiles/bxk6ngnt.default/Mail/pop.att.yahoo.com/Inbox. These are unique files that contain folders.
right click on your account on the left and select new folder.
-
How to write the JTables Content into the CSV File.
Hi Friends
I managed to write the Database records into the CSV Files. Now i would like to add the JTables contend into the CSV Files.
I just add the Code which Used to write the Database records into the CSV Files.
void exportApi()throws Exception
try
PrintWriter writing= new PrintWriter(new FileWriter("Report.csv"));
System.out.println("Connected");
stexport=conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_UPDATABLE);
rsexport=stexport.executeQuery("Select * from IssuedBook ");
ResultSetMetaData md = rsexport.getMetaData();
int columns = md.getColumnCount();
String fieldNames[]={"No","Name","Author","Date","Id","Issued","Return"};
//write fields names
String rec = "";
for (int i=0; i < fieldNames.length; i++)
rec +='\"'+fieldNames[i]+'\"';
rec+=",";
if (rec.endsWith(",")) rec=rec.substring(0, (rec.length()-1));
writing.println(rec);
//write values from result set to file
rsexport.beforeFirst();
while(rsexport.next())
rec = "";
for (int i=1; i < (columns+1); i++)
try
rec +="\""+rsexport.getString(i)+"\",";
rec +="\""+rsexport.getInt(i)+"\",";
catch(SQLException sqle)
// I would add this System.out.println("Exception in retrieval in for loop:\n"+sqle);
if (rec.endsWith(",")) rec=rec.substring(0,(rec.length()-1));
writing.println(rec);
writing.close();
}With this Same code how to Write the JTable content into the CSV Files.
Please tell me how to implement this.
Thank you for your Service
JofinHi Friends
I just modified my code and tried according to your suggestion. But here it does not print the records inside CSV File. But when i use ResultSet it prints the Records inside the CSV. Now i want to Display only the JTable content.
I am posting my code here. Please run this code and find the Report.csv file in your current Directory. and please help me to come out of this Problem.
import javax.swing.*;
import java.util.*;
import java.io.*;
import java.awt.*;
import java.awt.event.*;
import javax.swing.table.*;
public class Exporting extends JDialog implements ActionListener
private JRadioButton rby,rbn,rbr,rbnore,rbnorest;
private ButtonGroup bg;
private JPanel exportpanel;
private JButton btnExpots;
FileReader reading=null;
FileWriter writing=null;
JTable table;
JScrollPane scroll;
public Exporting()throws Exception
setSize(550,450);
setTitle("Export Results");
this.setLocation(100,100);
String Heading[]={"BOOK ID","NAME","AUTHOR","PRICE"};
String records[][]={{"B0201","JAVA PROGRAMING","JAMES","1234.00"},
{"B0202","SERVLET PROGRAMING","GOSLIN","1425.00"},
{"B0203","PHP DEVELOPMENT","SUNITHA","123"},
{"B0204","PRIAM","SELVI","1354"},
{"B0205","JAVA PROGRAMING","JAMES","1234.00"},
{"B0206","SERVLET PROGRAMING","GOSLIN","1425.00"},
{"B0207","PHP DEVELOPMENT","SUNITHA","123"},
{"B0208","PRIAM","SELVI","1354"}};
btnExpots= new JButton("Export");
btnExpots.addActionListener(this);
btnExpots.setBounds(140,200,60,25);
table = new JTable();
scroll=new JScrollPane(table);
((DefaultTableModel)table.getModel()).setDataVector(records,Heading);
System.out.println(table.getModel());
exportpanel= new JPanel();
exportpanel.add(btnExpots,BorderLayout.SOUTH);
exportpanel.add(scroll);
getContentPane().add(exportpanel);
setVisible(true);
public void actionPerformed(ActionEvent ae)
Object obj=ae.getSource();
try {
PrintWriter writing= new PrintWriter(new FileWriter("Report.csv"));
if(obj==btnExpots)
for(int row=0;row<table.getRowCount();++row)
for(int col=0;col<table.getColumnCount();++col)
Object ob=table.getValueAt(row,col);
//exportApi(ob);
System.out.println(ob);
System.out.println("Connected");
String fieldNames[]={"BOOK ID","NAME","AUTHOR","PRICE"};
String rec = "";
for (int i=0; i <fieldNames.length; i++)
rec +='\"'+fieldNames[i]+'\"';
rec+=",";
if (rec.endsWith(",")) rec=rec.substring(0, (rec.length()-1));
writing.println(rec);
//write values from result set to file
rec +="\""+ob+"\",";
if (rec.endsWith(",")) rec=rec.substring(0,(rec.length()-1));
writing.println(rec);
writing.close();
catch(Exception ex)
ex.printStackTrace();
public static void main(String arg[]) throws Exception
Exporting ex= new Exporting();
}Could anyone Please modify my code and help me out.
Thank you for your service
Cheers
Jofin -
How to get Document Set property values in a SharePoint library in to a CSV file using Powershell
Hi,
How to get Document Set property values in a SharePoint library into a CSV file using Powershell?
Any help would be greatly appreciated.
Thank you.
AA.Hi,
According to your description, my understanding is that you want to you want to get document set property value in a SharePoint library and then export into a CSV file using PowerShell.
I suggest you can get the document sets properties like the PowerShell Command below:
[system.reflection.assembly]::loadwithpartialname("microsoft.sharepoint")
$siteurl="http://sp2013sps/sites/test"
$listname="Documents"
$mysite=new-object microsoft.sharepoint.spsite($siteurl)
$myweb=$mysite.openweb()
$list=$myweb.lists[$listname]
foreach($item in $list.items)
if($item.contenttype.name -eq "Document Set")
if($item.folder.itemcount -eq 0)
write-host $item.title
Then you can use Export-Csv PowerShell Command to export to a CSV file.
More information:
Powershell for document sets
How to export data to CSV in PowerShell?
Using the Export-Csv Cmdlet
Thanks
Best Regards
TechNet Community Support
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact
[email protected] -
HT201066 How can I split a long recording audio file into multiple audio files?
Hi,
How can I split a long recording audio file into multiple audio files?
With Windows I was using Nero software, which is not available for Mac. Is there any similar app for free for Mac?
Thanks!Hello ingiorgio
You can import the track into GarageBand and then split the tracks there. Once that it is done you can highlight the ruler section and then share the song to iTunes to get them separated out.
GarageBand - Split regions in the Tracks area
http://help.apple.com/garageband/mac/10.0/#gbnd76fcce04
GarageBand - Share songs to iTunes
http://help.apple.com/garageband/mac/10.0/#gbndfb96a96f
Regards,
-Norm G. -
How To Split Large Excel or CSV Files into Smaller Files
Does anyone know how to split a large Excel or CSV file into multiple smaller files? Or, is there an app that will work with Mac to do that?
split [-a suffix_length] [-b byte_count[k|m]] [-l line_count] [-p pattern] [file [name]]
is a native Terminal command. Read up more on https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/ man1/split.1.html
I prefer to use gSplit which is gnu coreutils.
You can install gnu coreutils using homebrew. -
Split MP3 file into multiple MP3 files
I've been through the help section and is till cant seem to figure out how to split a large (2 hour) mp3 file into multiple mp3 files.
I have a 2 hour MP3 recording (Stereo not mono), and I have added markers (by simply pressing keyboard shortcut 'M' to add marker), but now I would like to know how to split that one mp3 files into multiple files from the markers which I have made...
All my marker types are "Cue" when i pressed 'M' at the locations I want the track to end/startYou didn't say which version of the program you're using, but in general, there is a way to do this by adding one additional cue at the very end of the file, then "merging" the cues (which converts the space between each cue into a "range"), and batch processing each range out to a separate file. (Make sure there's a cue at the beginning of the file, or at least the first material you want in a new file.)
Now that you've done all that work, I'm going to tell you that what you're doing is probably a bad idea. Opening any mp3 in Audition and resaving back to mp3 means you've doubled the losses and distortion inherent with the mp3 compression. Audition can't do lossless editing of mp3s. The bottom line: are you really splitting the file, and does the result have to be mp3 format? If saving to a (wave file, or any other lossless format, then what I'm saying doesn't matter.
In any case, "mp3 Direct Cut" is a terrific freeware program that does lossless mp3 editing, and I'd highly recommend you try it. It's saved my neck a number of times. See http://www.mpesch3.de. -
Script to merge multiple CSV files together with no duplicate records.
I like a Script to merge multiple CSV files together with no duplicate records.
None of the files have any headers and column A got a unique ID. What would be the best way to accomplish that?OK here is my answer :
2 files in a directory with no headers.
first column is the unique ID, second colomun you put whatever u want
The headers are added when using the import-csv cmdlet
first file contains :
1;a
2;SAMEID-FIRSTFILE
3;c
4;d
5;e
second file contains :
6;a
2;SAMEID-SECONDFILE
7;c
8;d
9;e
the second file contains the line : 2;b wich is the same in the first file
the code :
$i = 0
Foreach($file in (get-childitem d:\yourpath)){
if($i -eq 0){
$ref = import-csv $file.fullname -Header id,value -Delimiter ";"
}else{
$temp = import-csv $file.fullname -Header id,value -Delimiter ";"
foreach($line in $temp){
if(!($ref.id.contains($line.id))){
$objet = new-object Psobject
Add-Member -InputObject $objet -MemberType NoteProperty -Name id -value $line.id
Add-Member -InputObject $objet -MemberType NoteProperty -Name value -value $line.value
$ref += $objet
$i++
$ref
$ref should return:
id
value
1
a
2
SAMEID-FIRSTFILE
3
c
4
d
5
e
6
a
7
c
8
d
9
e
(get-childitem d:\yourpath) -> yourpath containing the 2 csv file
Maybe you are looking for
-
We have been compromised within our businesses from an empolyee that was terminated with GNU Emac Api enulation program that has infiltrated all of our brand new Apple Computers. We made a decision to move to Mac's this year because of all our busine
-
Can not upload from itunes10 Windows to ipod shuffle - error 50
I have an ipod shuffle 4th generation and am using itunes10 running on a PC Windows XP SP3. When synchronising the device from itunes, I always get the message <itunes is unable to copy "song" to ipod "name" because of an unknown error (-50). Irrespe
-
Cannot save alias. The alias cannot be saved at this time.
I'm trying to create an alias and cannot. Error is title of this post. I've searched this issue in the support community and only see similar issues as recent as 2013 and 2012. I'm wondering if there is any support for this issue. Thanks!
-
Problem with library and vault
I would like to know how to take all photos out of Aperture and start over. When I upgraded to V.3, I think I set things up incorrectly. I had a vault but now there is no vault and I now have folders that appear under Projects & Albums that indicate
-
App store and safari 6 conflict
app store and safari 6 conflict