Coldfusion 9 CFTransaction/CFQuery Timeouts

Question

I have a script that is pulling in data from an excel document using cfspreedsheet, performing data checks, and inserting the data into the appropriate tables. Each XLS document only has about 1000 records. The scripts are wrapped in a CFTransaction tag to ensure rollback if the script errors.
When I run these scripts they will succeed if I'm only importing around 400 records or less. If I try to import any more than that I receive a 500 server error from IIS. I'm guessing that this is some sort of timeout issue. In order to try and force the success of the script, I upped the requesttimeout setting to 900000. This did not solve the problem.
Can anyone tell me how to troubleshoot and/or fix this problem?
<cfsetting                    requesttimeout="900000"                    showdebugoutput="true"                    enablecfoutputonly="false"          />

Answer 1

Good to hear from you Bob!
Merging my sql statements together is  exactly what I have been doing the last 24 hours, unfortunately without luck.
For inserts, I have used this syntax:
INSERT INTO table (column_1,column2) VALUES
('value 1 for column 1','value 1 for column 2')
,(value 2 for column 1','value 2 for column 2')
,(value 3 for column 1','value 3 for column 2')
..etc
The problem is that the majority of my queries are update queries, so I changed my DSN to be able to handle multiple queries within the same <cfquery> by adding "allowMultipleQuery=true" in the text box "connection string" under advanced settings for my DSN.
Then I merged chunks of 250 update statements together throughout my loop (of about 6000 rows), and then outputted them in a <cfquery>, then starting on a new chunk and so on. This decreased the number of queries vastly off course, but my request ran even slower.
I produce the sql statements as strings (off course), and therefore I can't use any cfqueryparams, but that's ok. I read that they could cause memory leaks in some situations, so in this case, I was only glad to be able to remove them (I ALWAYS use them otherwise, to avoid sql injection off course).
I should also mention that I have a cftransaction around every 250 rows, that commits the insert and update queries, throughout the loop (every 250th row). From the beginning I had the transaction around the entire loop, which worked pretty well up to about 5000 rows.
Now I'm looking into memory leak issues. One guy wrote that he had a similar project like ours, but had problems with the memory usage, so he said he wrote a function that imported a part of the content, then "did a gateway call to the CFC" at the end, to the same function, with an argument that told the function to import the next part and so on. By doing this he said that the memory was flushed during the import, between the different parts of the import, resolving his issue. You can read about it here (one of the replies)
http://stackoverflow.com/questions/2364682/memory-implications-of-returning-a-query-from-a -cfc
I'm not sure though what he means with "gateway call". If it has anything to do with event gateways? If so, I'm really lost. Never tried that.
Anyway, now I will try to run the cf debugger, and see how my queries are doing, hopefully it will bring some clarity. I'm in Sweden, so it's getting late. 10:14 PM.
/Jörgen