Spreadsheet String To Array DBL Performance Improvement
While developing and profiling a batch data-analysis program, I found that the "Read From Spreadsheet File.vi" function was where the majority ofthe execution time was spent. Digging further, I found the time was spent in the "Spreadsheet String To Array" function. Obviously I cannot look inside this function, but I wondered if I can somehow speed it up. Attached is a Vi that illustrates what I came up with as an alternative.
Basically using "Spreadsheet String To Array" only to split it up to a 2D String Array, and then "Fract/Exp String to Number" to convert the strings to numbers.
It consistently runs in just over 60% of the time of using Spreadsheet String To Array on it's own to do the conversion.
I am curious as to why using the two functions is quicker than just the one on it's own? Perhaps NI can improve the internals a bit? Or perhaps it is not a fair comparison?
Labview 8.6f1 Win XP SP2
Message Edited by pauldavey on 03-25-2009 02:55 PM
Attachments:
Speed Test Spreadsheet String to Array.vi 26 KB
Adding a couple of other conversions to test against i noticed that:
1. Scan from string through a double loop was the slowest
2. Scan Value was slightly faster
3. Direct conversion through Spreadsheet to array as in OP
4. 2-step convert through Fract/Exp.
Could it be that Spreadsheet to array uses Scan Value internally?
/Y
LabVIEW 8.2 - 2014
"Only dead fish swim downstream" - "My life for Kudos!" - "Dumb people repeat old mistakes - smart ones create new ones."
G# - Free award winning reference based OOP for LV
Similar Messages
-
Spreadsheet string to array slow?
We have an apllication here where we would like to save all our data 120
files with 4M each and then read it back.
I ve changed now to zipped files, but reading from the disk seems not to
be the problem. I used 'profile' to analyze the time consumption. I
think it might be the 'spreadsheet string to array VI' which makes it so
slow, 1 second is a lot to sort 450k numbers. (BTW its a win2000 PC of
this year)
Any hints how to improve it?
Best Regards
Urs Böglihi,
I do this small program. Program generate, write and read array with double precision. I generated array with 1.000.000 rows and 10 columns. Read is fast in this program, because file is 'cached' in Windows Virtual Memory.
When you close LabVIEW and try read file with second VI on my PC I need 3 second for this array [on the disk have file 80MB].
It is programs in LabVIEW 7.0
Attachments:
readArrayFile.zip 186 KB -
Spreadsheet string to array notation
Hi all,
I am using the spreadsheet string to array function and basically my problem is as follows:
My inputs are in scientific notation so I use this in string format. The data is separated with a ",". What I get is just the argument rounded.
As I just realized I didn´t make myself clear I´ve attached a photo of the problem. Notice that probe 4 is the input from which I want to build the array, and probe 5 is the output. I just have the 9 but what happens with the rest of the number?
PD: I´ve tried many formats but none of them seems to work.
Thank for your help
Solved!
Go to Solution.
Attachments:
notation.png 145 KBYour system is set up to use a comma (,) as the decimal point? Try using "%.;%^.3e" for your format. The "%.;" tells the scan to use the period as the decimal point.
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines -
Como obtener un array dbl de un string de 2 bytes
como obtener un array dbl de un string de 2 bytes
Adjuntos:
dbl.png 37 KBHola, un string de dos bytes no puede contener un double ya que el dbl es un número de 8 bytes.
Acaso estás recibiendo alguna medicion de un conversor de 16 bits (un instrumento externo, un PIC o algo así)? Si es así el valor correspondiente en la unidad de medición (V, A, RPM...) debe calcularse con
bits recibidos / 65536 * máximo de la medición
Ejemplo: en un voltímetro de 100V / 16 bits, un valor de 8000 (bits) equivale a una medición de 12.207V
Proud to use LW/CVI from 3.1 on.
My contributions to the Developer Zone Community
If I have helped you, why not giving me a kudos? -
Req help in conversion of string to array of args
Hi,
i need help in converting a string into array of args just like command line args
i have a string s1 = 12 13 56 etc
now i want to copy that into an array a[]
a[0] = 12
a[1] = 13 etc
Thanks for helpU can use String tokenizer.Why prefer that over split?Yah Bobby u can use split too..
J2SE 1.4 added the split() method to simplify the task of breaking a string into substrings, or tokens.
Thanks BigDaddy.. I realized after you told.
try this too..
String str = "Your string";
String[] arr = str.split (" ");
for (int i=0; i < arr.length; i++)
System.out.println (arr); -
How to search for particular string in array?
I am struggling to figure out how to search array contents for a string and then delete the entry from the array if it is found.
The code for a program that allows the user to enter up to 20 inventory items (tools) is posted below; I apologize in advance for it as I am also not having much success grasping the concept of OOP and I am certain it is does not conform although it all compiles.
Anyway, if you can provide some assistance as to how to go about searching the array I would be most grateful. Many thanks in advance..
// ==========================================================
// Tool class
// Reads user input from keyboard and writes to text file a list of entered
// inventory items (tools)
// ==========================================================
import java.io.*;
import java.text.DecimalFormat;
public class Tool
private String name;
private double totalCost;
int units;
// int record;
double price;
// Constructor for Tool
public Tool(String toolName, int unitQty, double costPrice)
name = toolName;
units = unitQty;
price = costPrice;
public static void main( String args[] ) throws Exception
String file = "test.txt";
String input;
String item;
String addItem;
int choice = 0;
int recordNum = 1;
int qty;
double price;
boolean valid;
String toolName = "";
String itemQty = "";
String itemCost = "";
DecimalFormat fmt = new DecimalFormat("##0.00");
// Display menu options
System.out.println();
System.out.println(" 1. ENTER item(s) into inventory");
System.out.println(" 2. DELETE item(s) from inventory");
System.out.println(" 3. DISPLAY item(s) in inventory");
System.out.println();
System.out.println(" 9. QUIT program");
System.out.println();
System.out.println("==================================================");
System.out.println();
// Declare and initialize keyboard input stream
BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in));
do
valid = false;
try
System.out.print(" Enter an option > ");
input = stdin.readLine();
choice = Integer.parseInt(input);
System.out.println();
valid = true;
catch(NumberFormatException exception)
System.out.println();
System.out.println(" Only numbers accepted. Try again.");
while (!valid);
while (choice != 1 && choice != 2 && choice != 9)
System.out.println(" Not a valid option. Try again.");
System.out.print(" Enter an option > ");
input = stdin.readLine();
choice = Integer.parseInt(input);
System.out.println();
if (choice == 1)
// Declare and initialize input file
FileWriter fileName = new FileWriter(file);
BufferedWriter bufferedWriter = new BufferedWriter(fileName);
PrintWriter dataFile = new PrintWriter(bufferedWriter);
do
addItem="Y";
System.out.print(" Enter item #" + recordNum + " name > ");
toolName = stdin.readLine();
if (toolName.length() > 15)
toolName = toolName.substring(0,15); // Convert to uppercase
toolName = toolName.toUpperCase();
dataFile.print (toolName + "\t");
do
valid = false;
try
// Prompt for item quantity
System.out.print(" Enter item #" + recordNum + " quantity > ");
itemQty = stdin.readLine();
// Parse integer as string
qty = Integer.parseInt (itemQty);
// Write item quantity to data file
dataFile.print(itemQty + "\t");
valid=true;
catch(NumberFormatException exception)
// Throw error for all non-integer input
System.out.println();
System.out.println(" Only whole numbers please. Try again.");
while (!valid);
do
valid = false;
try
// Prompt for item cost
System.out.print(" Enter item #" + recordNum + " cost (A$) > ");
itemCost = stdin.readLine();
// Parse float as string
price = Double.parseDouble(itemCost);
// Write item cost to data file
dataFile.println(fmt.format(price));
valid = true;
catch(NumberFormatException exception)
// Throw error for all non-number input (integers
// allowed)
System.out.println();
System.out.println(" Only numbers please. Try again.");
while (!valid);
// Prompt to add another item
System.out.println();
System.out.print(" Add another item? Y/N > ");
addItem = stdin.readLine();
while ((!addItem.equalsIgnoreCase("Y")) && (!addItem.equalsIgnoreCase("N")))
// Prompt for valid input if not Y or N
System.out.println();
System.out.println(" Not a valid option. Try again.");
System.out.print(" Add another item? Y/N > ");
addItem = stdin.readLine();
System.out.println();
// Increment record number by 1
recordNum++;
if (addItem.equalsIgnoreCase("N"))
System.out.println();
System.out.println(" The output file \"" + file + "\" has been saved.");
System.out.println();
System.out.println(" Quitting program.");
while (addItem.equalsIgnoreCase("Y"));
// Close input file
dataFile.close();
if (choice == 2)
try {
Read user input (array search string)
Search array
If match found, remove entry from array
Confirm "deletion" and display new array contents
catch block {
} // class
// ==========================================================
// ListToolDetails class
// Reads a text file into an array and displays contents as an inventory list
// ==========================================================
import java.io.*;
import java.util.StringTokenizer;
import java.text.DecimalFormat;
public class ListToolDetails {
// Declare variable
private Tool[] toolArray; // Reference to an array of objects of type Tool
private int toolCount;
public static void main(String args[]) throws Exception {
String line, name, file = "test.txt";
int units, count = 0, record = 1;
double price, total = 0;
DecimalFormat fmt = new DecimalFormat("##0.00");
final int MAX = 20;
Tool[] items = new Tool[MAX];
System.out.println("Inventory List");
System.out.println();
System.out.println("REC.#" + "\t" + "ITEM" + "\t" + "QTY" + "\t"
+ "PRICE" + "\t" + "TOTAL");
System.out.println("\t" + "\t" + "\t" + "\t" + "PRICE");
System.out.println();
try {
// Read a tab-delimited text file of inventory items
FileReader fr = new FileReader(file);
BufferedReader inFile = new BufferedReader(fr);
StringTokenizer tokenizer;
while ((line = inFile.readLine()) != null) {
tokenizer = new StringTokenizer(line, "\t");
name = tokenizer.nextToken();
try {
units = Integer.parseInt(tokenizer.nextToken());
price = Double.parseDouble(tokenizer.nextToken());
items[count++] = new Tool(name, units, price);
total = units * price;
} catch (NumberFormatException exception) {
System.out.println("Error in input. Line ignored:");
System.out.println(line);
System.out.print(" " + count + "\t");
System.out.print(line + "\t");
System.out.print(fmt.format(total));
System.out.println();
inFile.close();
} catch (FileNotFoundException exception) {
System.out.println("The file " + file + " was not found.");
} catch (IOException exception) {
System.out.println(exception);
System.out.println();
// Unfinished functionality for displaying "error" message if user tries to
// add more than 20 tools to inventory
public void addTool(Tool maxtools) {
if (toolCount < toolArray.length) {
toolArray[toolCount] = maxtools;
toolCount += 1;
} else {
System.out.print("Inventory is full. Cannot add new tools.");
// This should search inventory by string and remove/overwrite matching
// entry with null
public Tool getTool(int index) {
if (index < toolCount) {
return toolArray[index];
} else {
System.out
.println("That tool does not exist at this index location.");
return null;
} // classData file contents:
TOOL 1 1 1.21
TOOL 2 8 3.85
TOOL 3 35 6.92Ok, so you have an array of Strings. And if the string you are searching for is in the array, you need to remove it from the array.
Is that right?
Can you use an ArrayList<String> instead of a String[ ]?
To find it, you would just do:
for (String item : myArray){
if (item.equals(searchString){
// remove the element. Not trivial for arrays, very easy for ArrayList
}Heck, with an arraylist you might be able to do the following:
arrayList.remove(arrayList.indexOf(searchString));[edit]
the above assumes you are using 1.5
uses generics and for each loop
[edit2]
and kinda won't work it you have to use an array since you will need the array index to be able to remove it. See the previous post for that, then set the value in that array index to null.
Message was edited by:
BaltimoreJohn -
Gluegen wrapper returning String or Array
How can I setup a wrapper function to return a String or
Array. In my wrapper.gg file I'd like to have something like:
public function mystrcat(str1:String, str2:String):String
char *ret = strcat(str1, str2);
return ret;
public function myarrayfunc():Array
int a[10];
return a;
Do I need to stuff the data in a ByteArray or something like
that?I've got trouble getting Strings to work within a gg wrapper. I tried adding the following within a gluegen file (note there are no pure c-code blocks, only the pseudo-AS):
public function getString():String {
return AS3_String("hello");
running gluegen with the following:
gluegen libharu.gg -oc hpdf.c -oas ../src/alchemy/hpdf/hpdf.as -cpackage cmodule.hpdf -package alchemy.hpdf -class hpdf
and the compiling using:
gcc hpdf.c -O3 -swc -o hpdf.swc
But all I get is errors about some conflicting gg_string type and warnings about incompatible pointer types?
ibharu.gg: In function 'impl_getString':
libharu.gg:11: warning: return from incompatible pointer type
libharu.gg: In function 'thunk_getString':
libharu.gg:10: warning: return makes pointer from integer without a cast
libharu.gg: At top level:
libharu.gg:12: error: conflicting types for 'gg_string'
libharu.gg:10: error: previous implicit declaration of 'gg_string' was here
However, if I change the code as follows everything compiles fine, and I'm able to use the produced swc in Flex:
public function getString():Array {
return AS3_Array("StrType", "hey there");
So basically, all I've done is returned an AS3_Array instead of AS3_String. Since my C-skills are limited and about 10 years old, I might be missing something very trivial here, but can anyone help me spot what it is? I'm compiling on OS X 10.5.7 -
Unexpected behavior of spreadsheet string to array function
Hello,
I found some weirdness in LabVIEW 2011 that I do not understand. In the attached vi, I provide a one-dimensional spreadsheet string separated by spaces. I use the spreadsheet string to array function to convert this spreadsheet string into an array of strings.
I ran into problems when I wanted to specify a space character as the delimiter.
The conversion works as intended, if I do not specify a delimiter (i.e., the default tab delimiter is used). But if I specify the delimiter, only the first element of the spreadsheet string is converted. I do not understand this behavior.
Thanks for your help.
Peter
Solved!
Go to Solution.
Attachments:
Test spreadsheet string to array.vi 9 KBYou don't have spaces in your string. You have tabs and of course the default delimiter works. Right click and select '\' Codes Display and see the \t.
-
Apparent inconsistency of "spreadsheet string to array"
System: LabView6.1, XPpro, 512MB, 2.4GHz
I have a sub-vi which I call to read a single column of data from a large CSV file. Files can be >500000 rows with 4-8 cols (ie. 15-30MB). The sub-vi uses "Read File", with byte stream type unwired, to return a string. This string is then passed to "spreadsheet string to array" (SStA), to obtain a 2D array. "Index Array" is then used to obtain the required data column.
If I wire the format string of SStA with %f, and the array type with a 2D double, the sub-VI takes about 8 seconds every time I call it.
If I wire format string with %s, and array type with 2D string (and convert to a double later, after index array), the sub-VI takes over 50s the first time it is c
alled, but only about 1s on each subsequent call.
The bottle neck is ONLY at SStA. I am not explicitly preallocating any arrays.
Can anyone explain what LabView is doing internally with respects to memory allocation? Why is the second routine slow on the first call only, and subsequently considerably faster than the first routine? Is there any alternative way of "priming" the second sub-vi so it is not slow on the first call? I guess reading only subsets of data at a time might help, but I'd still like to understand what is going on with the current approach.Roo Payne wrote:
> Can anyone explain what LabView is doing internally with respects to
> memory allocation? Why is the second routine slow on the first call
> only, and subsequently considerably faster than the first routine? Is
> there any alternative way of "priming" the second sub-vi so it is not
> slow on the first call? I guess reading only subsets of data at a time
> might help, but I'd still like to understand what is going on with the
> current approach.
Basically NI seems to have added some performance optimization with
internal caching to the Spreadsheet String to Array function in the case
of String outputs. Without it it would always take 50 seconds. If the
input string does change significantly it probably won't be such a huge
speed up anymore.
For numeric type outputs no caching has been added and wouldn't help
that much in comparison.
Rolf Kalbermatter
Rolf Kalbermatter
CIT Engineering Netherlands
a division of Test & Measurement Solutions -
Issues with spreadsheet string to array
Hi, I have a spreadsheet string (tab delimited) and I am trying to convert it into an array of double with 6 digits of precision after the decimal point. However, as shown in the attached example, the function "spreadsheet to array" is not giving me the desired digit of precision (only 2 after decimal point). I have tried almost everything and I am frustrated with this function. CAN SOMEONE PLZ HELP ME ??
Attachments:
Block diagram.png 13 KB
String to array double.vi 10 KB
Panel.png 28 KBI have no LV 8.x installed, so I can't check your Code. But it might be your settings for the indicator. In LV 7.1 it defaults to 6 significant digits, that means the reading is 12345,6789 but (only) your indicator displays 12345,6.
Right click on your indicator (Front Panel) and go to formatting.
Felix
www.aescusoft.de
My latest community nugget on producer/consumer design
My current blog: A journey through uml -
Delimiter for the Spreadsheet string to array
Hello,
I would like to use the Spreadsheet string to array function to process the data obtained from reading a text file that has 2 columns separated by at least one space character. Can the spreadsheet string to array have delimiters other than a comma or tab?
Regards,
KasparHere's an example of what I had in mind (LabVIEW 8.0). (The delimiter constants are displayed in "'\'-codes").
As you can see, it ignores consecutive delimiters (even empty lines )
Message Edited by altenbach on 06-29-2006 09:10 AM
LabVIEW Champion . Do more with less code and in less time .
Attachments:
DirtySpreadsheetStringToArray.png 17 KB
DirtySpreadsheetStringToArray.vi 14 KB -
Converting from spreadshet string to array and then back to spreadsheet string
My questions is; why is the Spreadsheet string to array function creating more data than the original string had when you change the array back into a spreadsheet string. Im trying to analyze a comma delimited file using array functions since my column and row size is constant, but my data varies. Thus my reason for not using string parsing functions which would get more involved and difficult. So, however, after i convert to a 2D array of data from the comma delimited file I read from, and then I convert back to string using the Array to Spreadsheet String, I get added columns to the file, which prevents another program from receiving these files. Also, the data which I am reading is not all contiguous, it has gaps in some places for empty data. Looking at the file compared to the original after it has gone from string to array and then back to string again, looks almost identical except for the file size which got larger by 400 bytes and where the original file has empty spaces, the new file has a lot of commas added. Any idea?
CharlesThe result you get is normal when the spreadsheet string contains rows of uneven length. Since the array rows have the same number of elements, nil values are added during the coonversion. And of course, the back to string conversion keep those added values in the string, with the associated commas.
example : 3 x 3 array
1,2,3
4
5,6,7
is converted into
1 2 3
4 0 0
5 6 7
then back to
1,2,3
4,0,0
5,6,7
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left... -
Creating String frm new String(charBuffer.array()) Vs charBuffer.toString()
Whats the difference in creating String from CharBuffer by using array and by using toString() ?
When ever i have some UTF-8 chars in my file (""someFile"), String created from new String( charBuffer.array()) appends some extra null/junk charaters at the very end of the file.
How ever when i try charBuffer.toString() its working fine.
For simple ASCII i.e ISO-*** charset both methods are working fine.
Please see below code for reproducing. Here "someFile" is any text file with some UTF-8 characters.
public char[] getCharArray()
throws IOException
Charset charset = Charset.forName("UTF-8");
CharsetDecoder decoder = charset.newDecoder();
FileInputStream fis = new FileInputStream("someFile");
FileChannel channel = fis.getChannel();
int size = (int) channel.size();
MappedByteBuffer mbb = channel.map(FileChannel.MapMode.READ_ONLY, 0 , size);
CharBuffer cb = decoder.decode(mbb);
channel.close();
fis.close();
return cb.array();
public String getAsString()
throws IOException
Charset charset = Charset.forName("UTF-8");
CharsetDecoder decoder = charset.newDecoder();
FileInputStream fis = new FileInputStream("someFile");
FileChannel channel = fis.getChannel();
int size = (int) channel.size();
MappedByteBuffer mbb = channel.map(FileChannel.MapMode.READ_ONLY, 0 , size);
CharBuffer cb = decoder.decode(mbb);
channel.close();
fis.close();
return cb.toString();
String fromToString = getAsString();
String fromCharArray = new String(getCharArray());Whats the difference in creating String from CharBuffer by using array and by using toString() ?array() returns the entire backing array regardless of offset and position. toString() takes those into account.
When ever i have some UTF-8 chars in my file (""someFile"), String created from new String( charBuffer.array()) appends some extra null/junk charaters at the very end of the file.More probably you haven't filled the array.
How ever when i try charBuffer.toString() its working fine.So there you go. -
How to use type cast change string to number(dbl)?can it work?
how to use type cast change string to number(dbl)?can it work?
Do you want to Type Cast (function in the Advanced >> Data Manipulation palette) or Convert (functions in the String >> String/Number Conversion palette)?
2 simple examples:
"1" cast as I8 = 49 or 31 hex.
"1" converted to decimal = 1.
"20" cast as I16 = 12848 or 3230 hex.
"20" converted to decimal = 20.
Note that type casting a string to an integer results in a byte by byte conversion to the ASCII values.
32 hex is an ASCII "2" and 30 hex is an ASCII "0" so "20" cast as I16 becomes 3230 hex.
When type casting a string to a double, the string must conform the the IEEE 32 bit floating point representation, which is typically not easy to enter from the keyboard.
See tha attached LabView 6.1 example.
Attachments:
TypeCastAndConvert.vi 34 KB -
Parse string into array for comparison
Hi All,
I am trying to parse a string message into an array of numbers so I can check if my event was successful.
\FF\FE\01\FD\02\00\00\00
I know I just need to read through and discard the \, but I don't know how to do that in Labview. I just need to check if the 6th byte = 00 or not.
Thanks! I mostly need help with parsing in labview.Assuming this is a plain ASCII string containing the letters 0..F and "\" as delimiter, you can simply used "Spreadsheet string to array" with the following settings:
Type= 1D U8 array
format= %x
delimiter= "\"
One problem is the extra delimiter at the beginning, so use array subset to skip the first element.
(EDIT: Ahh, Darin beat me with a similar solution. Mine's a little easier ).
Message Edited by altenbach on 08-06-2009 04:30 PM
LabVIEW Champion . Do more with less code and in less time .
Attachments:
StringToArray.png 13 KB
Maybe you are looking for
-
SQL Loader step hangs in unix script
Hi, We have a nightly job that separately truncates 6 tables and then loads them with flat file data. It works fine almost always. Eight months ago and then yesterday, one of the 6 load steps(not the same table) hung up and did nothing. Reruns ran fi
-
Can Queries with varients be precalculated ?
Hello All, How to precalculate a Query or Web templet which has input variable's. Actually i am following this process for precalculation, Eg: Query> Publish>BEx Broadcast> new settings> Precalculation settings...... I have a query which requires inp
-
When I try to update iTunes to 11.1.0, I can't due to error C2QK82VCDTY3
My Macbook is updated with the latest version of every software but iTunes. However, when I try to update it, I can't due to error C2QK82VCDTY3. Can anyone help me? Thanks, Giacomo
-
I would like to make a photo I have available for others to view on the web. How?
I have a photo that I would like a company to see so they might help me find a duplicate. Is this possible?
-
Why does a ? get put in the title bar?
When this page loads, a question mark (?) appears on the title page. Anyone know why? http://web.me.com/tjkk1/JR/Blog/Blog.html Thank you!