CF9 and SOLR indexing

Question

We are using CF9 64-bit and setting up a SOLR collection for an HR application. The database contains several million records and includes resumes that we want to do full text searches on.
We started out by using cfindex to create the index but it would bomb out after just a few thousand records with an error about "warming threads" (I don't have the exact error handy but can get it later) and the indexing would have to be manually restarted. This wasn't a good solution for a multi-million record operation..
Next, we created a custom Data Import Handler (DIH) outside of CF using the instructions in the SOLR wiki. This index worked great and was very fast. However, the ColdFusion tags (cfsearch, etc.) would not work with this index. We even made sure to duplicate the required nodes (<custom1> <custom2>, etc.) that the cfindex tag would have created. Still cannot search that index.
We'd really rather not reinvent the wheel and have to write custom search code. Obviously, we like using CF and it would be great if we can use the built-in indexing and searching capability.
Any ideas on how we can either 1) make the <cfindex> work without stopping OR 2) go ahead and use the custom DIH and be able to make the <cfsearch> work properly?
Dana

Answer 1

I only have just over 500 records that I am trying to index, which they do consist of some large documents, and I try to loop through using the cfindex and I also get this error:
Error_opening_new_searcher_exceeded_limit_of_maxWarmingSearchers4_try_again_later
I found that if I put this in my loop
<cfscript>
    thread = CreateObject("java", "java.lang.Thread");
    thread.sleep(1000);
</cfscript>
then I no longer have the error, but it does take a long long time to index.  I also would like a better solution.
The coldfusion debugger shows that it is erroring out on the custom4 field.  I don't know if the custom fields are struggling more than the main body field.  Anyway, I am continuing to research my options.