Performance issues when working with huge lists

Question

I’ve got a script that reads a large CSV spreadsheet
and parses the data into a list of the form [[A1,B1,C1],
[A2,B2,C2], [A3,B3,C3]] and a second list of the form
[#A1:B1,#A2:B2,#A3:B3] etc… The actual spreadsheet is about
10 columns x 10,000 rows. Reading the file string goes fast enough,
the parsing starts off fast but slows to a crawl after about 500
rows (I put the row count on the stage to check progress). Does
anyone know if the getaProp, addProp, and append methods are
sensitive to the size of the list?
A sample of one of the parsing loops is below. I’m
aware all interactivity will stop as this is executed. This script
is strictly for internal use, it crunches the numbers in two
spreadsheets and merges the results to a new CSV file. The program
is intended to run overnight and the new file harvested in the
morning.

Answer 1

> Does anyone know if the getaProp, addProp, and
> append methods are sensitive to the size of the list?
Is this a trick question? Sure they are. All of them.
Addprop and append are quite fast (due to the list object
scalable
preallocating memory as required), so i doubt that they are
the cause of
the problem.
GetAProp will search each item in the list, therefore, if you
are
searching for the last item, or if the item is not in the
list, the more
the items, the slower the command.
Didn't go through all your code but I noticed
- this: repeat with rowCount = 2 to file2string.line.count
Big no-no! Line counting is a very slow operation for it to
be a
evaluated in a loop.
- and this: myFile2data.append(myLineData)
String operations like this require memory reallocation, and
therefore
are very slow. If you do conclude that such an operation
causes the
problem, consider using a preallocated buffer (create a big
string in
advance) and then use
mydata.char.[currentoffset..(currentoffset+newstr.length)]=newstr
This can make code run even hundreds times faster, compared
to the
append method.
Applied CD wrote:
> I?ve got a script that reads a large CSV spreadsheet and
parses the data into a
> list of the form [[A1,B1,C1], [A2,B2,C2], [A3,B3,C3]]
and a second list of the
> form [#A1:B1,#A2:B2,#A3:B3] etc? The actual spreadsheet
is about 10 columns x
> 10,000 rows. Reading the file string goes fast enough,
the parsing starts off
> fast but slows to a crawl after about 500 rows (I put
the row count on the
> stage to check progress). Does anyone know if the
getaProp, addProp, and
> append methods are sensitive to the size of the list?
>
> A sample of one of the parsing loops is below. I?m aware
all interactivity
> will stop as this is executed. This script is strictly
for internal use, it
> crunches the numbers in two spreadsheets and merges the
results to a new CSV
> file. The program is intended to run overnight and the
new file harvested in
> the morning.
>
> writeLine("File 2 Data Parsing" & RETURN)
> myOrderColumn = myHeaders2.getOne("OrderNum")
> myChargesColumn = myHeaders2.getOne("Cost")
> myFile2data = []
> mergedFedExCharges = [:]
> repeat with rowCount = 2 to file2string.line.count
> myLineData = []
> repeat with i = 1 to
file2string.line[rowCount].item.count
> myItem = file2string.line[rowCount].item
> if i = 1 then
> myItem = chars(myItem,2,myItem.length)
> end if
> myLineData.append(myItem)
> end repeat
> if myLineData.count = myHeaders2.count then
> myFile2data.append(myLineData)
> myOrderSymbol = symbol("s" &
myLineData[myOrderColumn])
> myCurrentValue =
getaProp(mergedFedExCharges,myOrderSymbol)
> if voidP(myCurrentValue) then
> mergedFedExCharges.addProp(myOrderSymbol,0.00)
> end if
> mergedFedExCharges[myOrderSymbol] =
mergedFedExCharges[myOrderSymbol] +
> value(myLineData[myChargesColumn])
> writeUpdate(myLineData[1])
> else
> writeError("file 2 : " & string(myLineData) &
RETURN)
> end if
> end repeat
>