CF8 Verity Indexing

I've been using Verity to build collection of data from my
database systems for several years now using CF5 and CF6.1. I
recently upgraded my system to CF8. While searching these
collections is 5 to 6 times faster, indexing these collections is
SIGNIFICANTLY slower.
My system has some 50+ screens that I use to collect and
store data into either MSAccess or SQL Sever databases. There are
26 different collections that I use to store this data in. After
the data from each screen is stored in the datbase, I then update
the pertinent collection based upon the screen and database
table(s) where the data is being stored. The collections are a
compromise between number of screens and tables. This enables users
to always be searching the most current data. Every night we run a
scheduled task to rebuild all of the collections because this
continual add/dropping of data during the day chops up the Verity
collections.
I noticed two thing after creating the new CF8 collections:
Indexings is 2 to 3 times slower that it was in CF8. Searching is 5
to 6 time faster than it was. I'm wondering two things at this
point:
1. Is this a common experience amoung CF6 to CF8 users?
2. Is there some different technique that you have to use to
index collections using CF8 that will improve the indexing speed?
Thank you in advance for your insight in this matter.
Len

No work can be performed by our development group without a complete analysis, project scope, project plan and project budget is developed. I wish I could simple pull CF9 into the shop or the Solr download but that is not possible unless all the technical aspects are provide ahead of time and submitted for approval in the form of a project plan.
I presume you have a computer at home? Why don't use use that? That's what I'm doing, to test your code!
Or do you stop being a developer at 5pm?
Thank you for offering to test Solr for me, it is appreciated.
Attached is a form we use provided by the IRS. When running CFINDEX and then CFSEARCH the document does not appear on the search results.
Thanks for your help.
NP, but there was nothing attached.
Oh, and sorry to take so long to reply... Xmas and all that bullsh!t got in the way.
Adam

Similar Messages

CF8 verity and acrobat 9.0 errors

we have an issue where the CF8 verity engine is not indexing
PDF files created with acrobat 9.0
the same PDF files generated with acrobat 8.1 work fine.
the acrobat 9.0 PDF files generate these errr messages in the
verity logs
Tue Jan 13 14:44:44 2009
msg(1): Warn E0-1514 (Drvr): TstrIOFilter:flt_kv: KV failed
on filtering document: error = 8
Tue Jan 13 14:44:44 2009
msg(1): Warn E2-0527 (Document Index): Document 3
(d:\developers\MRC\wesinfo\version2\admin\test\wespress_testing\docs\Stark4_acrobat_9_vie w_7.pdf):
Stream error (-140) - SKIPPING
are there a new set of filters available for verity, so that
it can index acrobat 9.0 files ?
also, we found that while cf8 verity can index word 2007
.docx files, it does not pickup the Title metatag.

I have figured it out....
Adobe Acrobat v8 saves PDFs in v5 (1.4) format by default. Adobe Acrobat v9 saves PDFs in v7 (1.6) format by default.
ColdFusions CFINDEX doesn't like PFDs saved in v6 (1.5) or higher. So if you upgrade to Adobe Acrobat v9 and want to use the PDFs in a CF verity search you need to configure Distiller to save in v5 (or earlier) format so they can be indexed. (Or just resave the files to be optimised for v5 after they have been created.)
My solution was to configure the application to do a CFFILE READ on the PDF when a user uploads the document into the system. The first 8 characters of the file contents indicate the PDF version. v4 and v5 will work. v6+ won't work.
<cfif Mid(fileContents, 6, 3) lte 1.4>

<cfelseif Mid(fileContents, 6, 3) gt 1.4>

</cfif>
Hope this comes in handy for someone one day!

Coldfusion MX7 Verity index

Hi
I have an older verision of CF. MX7 7,0,2. It is running great and supports all of our users.
The one issue I recently noticed is that the verity collections do not index any document pass adobe 6.
I researched this issue and found that anything later than 6.0 apears to be unsupported in the MX7 verity indexing.
Out of 3000 documents I am missing 800 that are newer and won't index due to being unsupported file types.
Does anyone know if there is an update or fix to allow MX7 to index adobe files later than version 6.0
I hate to buy a whole new version of CF, but may have to.
---- bryan

Well. CFMX7 is out of support now, so there aren't going to be any more hotfixes, unless something really really dire comes up by way of security hole. But you really ought to have already migrated upwards.
And Verity doesn't exist at all as far as "live" software goes. It's dead. And has been since 2005 (before PDF v6.0 came out).
So I don't like your chances.
That said, depending on the architecture, the document filters might be third-party extensions so perhaps subsequent versions of CF might have filters for later versions of CF, even if it's the same old "death knell" version of Verity shipped with CF8 & CF9 (it won't be shipped with CF10 at all).
All the hotfixes for CFMX7 are listed here:
http://kb2.adobe.com/cps/178/tn_17883.html
I didn't look through them, but you can see if they mention anything.
CF8 is a much better product than CFMX7, much much much faster if nothing else. CF9 is an improvement again of CF8 obviously. You should upgrade.
Adam

CF8 Verity K2 server excessive memory

I can see this is an old post, but it still appears to be
relavent today even with CF8. My understanding is that CF8 is using
Verity K2 5.5 (same as CF7). I am able to take up a massive amount
of system memory just by simply adding collections without even
indexing them. I noticed this problem when we started getting ready
deploy several of our home grown CMS products onto our development
server and I noticed that a large chunck of memory was being taken
up.
So, I decided simply to see what the server would do if I
just started adding a bunch of empty verity collections. The memory
taken up by just 60 collections was over 1G. I tried to take it to
120 collections, but verity kept crashing (out of memory errors).
I read in this post (
Verity
Not Working in CF MX 7) that it was possible to seperate out
the verity server from the CF server, so I have been attempting to
do just that. Rob Rush provides a good informational link in that
post. So far I have been unsuccessful. I have followed that
instructions in the CF documentation and from the technote and it
seems like I have the two servers communicating, but when I go to
perform an index, it fails.
If anyone has successfully seperated the servers, it would be
of great interested and help to me if you can shed any light on
this.

Thanks for the suggestion. I tried just re-installing the CF8
Search server, but it still didn't work so I uninstalled it and
then reinstalled it. After that I was able to create the new
collections and finally, re-index the new collections. All seems
well for now. I have four more systems to recreate the collections
for and then re-index all of them.
This whole thing begs the questions:
1. How long is this installation going to survive?
2. Is this an indication of a very fragile product?
3. Are you and I the only ones having problems?
4. Are you and I the only ones using CF8 Verity?
5. Is Adobe aware of any problems with CF8 Verity?
6. Is there something wrong with my code that has worked
successfully with CF4, CF5 & CF6 now that I'm using CF8?
I’m sure there are other questions that should be asked
but I’m too tired and hungry to think about them right now.
Thanks again for your insight.
Len

CF8 verity french reports wrong document count

I had to rebuild a number of collections that use the verity
language packs in migtrating to 8. All of the document counts
displayed in the cfadministrator and by calling #recordsSearched#
come out correctly EXCEPT for the french language one. I deleted
and rebuilt the collection 3 times, same results each time. As near
as I can tell, the collection is behaving correctly in terms of
finding items- no duplicates displayed, and the most I can find
with any search is 55 documents (expected total)
Since I am pulling from a database, I know exactly how many
records I have (76), 55 of which have electronic versions and get
indexed. I looped a query to index the documents and output the
title and action each time to visually verifiy that it looped 76
times, and did the indexing 55 times. Yet my document count is 133
with each build.
Has anyone else seen anything similar? It's not a fatal
problem by any means, but it is weird and I would love to know that
I can rely on the recordSearched returned. (Arabic, Spanish,
Russian, Portuguese, English were all fine.)

CF_Ninja,
I get your point. What you discovered isn't new, and didn't start with Adobe. In fact, it is so common in commercial software, you can even call it a new Murphy's law. If version s is the latest version of a given software(say, Coldfusion 9), and there has been a switch from module m1 in version s-1 to module m2 in version s (say, the switch from Verity in CF8 to Solr in CF9), then the module m1 will invariably be crappy.
The developers of CF8, already with a euphoric eye on the future use of Solr, would inadvertently have ignored any further development of Verity. It's only human to think: Why waste all that effort on an end-of-life product?
I share your choice. The Coldfusion Jedi has a good comparison of Verity and Solr

CF9 and Verity indexing error - Linux 64

I am running CF 9.0.1 Standard Edition on an openSUSE 11.3 64-bit server, with Apache 2.2.15.
I am trying to create a verity collection on the server. The service is running as the collection is created without a problem, but when I attempt to create the index, either through the administration interface or with createindex, I receive the following errors in the sysinfo.log file.
msg(1): Error   E0-0720 (I/O Filter): Could not load filter 'flt_kv -recognize -bifmime' which is named in your style.uni file.
Thu Mar 10 09:44:25 2011
msg(1): Warn    E2-0527 (Document Index): Document 8899 (/export/www/htdocs/lter/googlee5473098f17d334b.html): Stream error (-2) - SKIPPING
Thu Mar 10 09:44:25 2011
msg(1): Error   E0-0720 (I/O Filter): Could not load filter 'flt_kv -recognize -bifmime' which is named in your style.uni file.
Thu Mar 10 09:44:25 2011
msg(1): Warn    E2-0527 (Document Index): Document 8900 (/export/www/htdocs/lter/data.cfm): Stream error (-2) - SKIPPING
The log file contains an entry for every document that it attempted to index, and the index remains empty.
Searches of the problem have not been helpful. I can not seem to find any recent instances of this problem.

Hi,
Look like Verity does not recognise the flt_kv_bifmime in the style.uni file. Check the style.uni and add the mime type ref.
Switch on Verity logging and check the apache log files for poss related errors.
The apache error ../lter/googlee*.html will be related to mime type as well prob same issue as above hence the ../lter/data.cfm is not being processed.
As the service is running it would be worth creating a manual test spider script to crawl / index the target files to get a confirmatory error message. The format is straightforward and examples can be found in and amongst the Verity K2 docs on Adobe Live Docs for CF 9 [ http://help.adobe.com/en_US/ColdFusion/9.0/Admin/WSc3ff6d0ea77859461172e0811cbf364104-7fb2 .html ]
If possible consider changing the search to SOLR as Verity K2 is no longer supported and Apache SOLR is the replacement. Again check the CF 9 docs and you can migrate from Verity to SOLR through the CFIDE/administrator.
Hope this helps.

Verity Index problem

Hi folks. I am trying to populate a verity collection from a
query but having no luck as best I can tell from the searchs that
come back null. So I created a collection, and can see it's there
using <cfcollection action="list"> And then I setup my query
as I need using:
<cfquery datasource="#DS#" name="getAbs">
select * from Abstracts
</cfquery>
and then define a new query with the key field (tmpkey)
defined to the path & file name I need:
<cfset qryTmp =
querynew("tmpkey,filename,title,author,year,affiliation")>
<cfoutput query="getAbs">
<cfset tmp = queryaddrow(qryTmp)>
<cfset tmp =
querysetcell(qryTmp,"tmpkey","e:\web\sctweb\abstracts\" &
#year# & "\" & #filename#)>
<cfset tmp = querysetcell(qryTmp,"filename",filename)>
<cfset tmp = querysetcell(qryTmp,"title",title)>
<cfset tmp = querysetcell(qryTmp,"author",author)>
<cfset tmp =
querysetcell(qryTmp,"affiliation",affiliation)>
</cfoutput>
If I CFDUMP TMPQRY everything looks great, all the fields
just as I want them. The I use CFINDEX to index my query like so:
<cfindex collection="sctabstracts"
query="qryTmp"
action="refresh"
type="file"
key="tmpkey"
>
But after that, I can't seem to return any results when I
search. Ideas?
Thanks
S

Thanks Sam, I've tried that as well and havn't had any luck.
Refresh is supposed to purge and then update so it seems like it
should work but.....
Thanks again though

CFMX 7 Verity Indexing Problem

I am having trouble indexing a collection on the PROD box, while it works fine with same code and same documents on the DEV box.
At first when user runs a search, error occurred where the cfindex key gave a sql syntax error ("WHERE 'key_id'=' ".).
Clearing the workspace folders in ws/ directory that verity uses helped rid of this error, but when collection deleted, and recreated, indexing it would not work anymore.
Now no errors or exceptions being thrown but search returns 0 results because the indexing failed (CFMX7 on verity bundled version, custom type cfindex).
Again, the same cfindex code and same text/documents recreate and reindex collections just fine on the DEV box. But, when attempted on the PROD box indexing returns nothing. Creating collection also does not work unless done manually on CF admin.
Any ideas please anyone?
thanks.

Upgrade your servers memory to at least 2gig. Indexing DB
tables is very memory intensive. Having read documentation from
Adobe when you index a DB you basically are indexing twice once to
gather the info from the tables and again to create the index from
the gathered info. Indexing directories of files cuts the time in
half.

Verity Indexes appear to Cross-Pollinate

We're seeing this issue crop up at various times where we
will call a simple cfsearch on one collection, and Verity will
return us data from another existing collection on the server.
We've done logging etc, and looking at the meta data of the
indexes, and all things look like they should (i.e. the number of
documents match that of what we'd expect there to be in that
collection). We are running cfmx 7.02 enterprise multi-server
install and have this application on it's own instance on a win2k3
box. The cfsearch is being called from w/in a cfc as:
quote:
<cfsearch collection="#arguments.collection#"
criteria="#arguments.args.q#" name="query1" type="natural"
maxrows="8193" />
Once I restart the ‘ColdFusion 7 Search Service’
in either beta (if we’re having problems there) or in
production, the Problem goes away – and the indexes correctly
return the data from the requested collection as they are supposed
to. Which leads me to believe that our application is not the
problem – but in the internals of how the ColdFusion cfsearch
tag API is calling the Verity K2 Server at certain periods. I still
have not nailed down exactly what the root cause is that makes call
through the cfSearch tag all of a sudden choose to use a different
index that what I passed it – but I do know that the restart
does re-align things.
Any help appreciated - we may have to open up a ticket on
this one

Well, after seeing this error come and go off and on - at
various times, I also see some logging in the
verity\Data\services\ColdFusionK2_server1\log\status.log at times
of '2008-08-12 06:00:30 Status: [Coll] Warning: collection xxxxx is
offline'
What can cause a collection to just go 'OFFLINE'?

Copying Verity Index

Since my shared hosting provider has the timeout set so low I
can't index my verity collection of pdf files before it times out.
So, I'm told I can index it locally, then FTP the
index/collection up to my site at the hosting provider.
But when I do this, the "url" variable in my search results
is messed up - actually reflects the absolute path on my local
machine!?
Is there a best practice way of copying a verity collection?
OR, an efficient way to index pdf files - I only have 118 of
them.
Thanks.

You can index locally on your machine and then upload the
collection and the files to your hosting, but the directory
structure (drive, folders, etc) must be the same in you local
machine and in your hosting provider for the files being indexed,
if not it won't work.
regards

Verity Indexing issue

Hi All,
Can anyone help with this one. We're running CF5 (migrating
to 8 soon, so please don't chuckle). However when we come to
reindex our current collections, it takes a matter of seconds and
returns with no error. I've tried this through CF Administrator and
through CFINDEX code that has been running fine for over 4 years.
It now just doesn't seem to acknowledge there are any files to
index.
Just FYI, we have around 10,000 topics so nowhere near the
limit, we're also installed on Unix. I've been checking permissions
for the 'nobody' user and it looks fine, I'm not sure how it could
have changed without manual intervention anyway.
This is working in our development server, but in production
it's not.
Any thoughts?
Kind regards
Paul Pounder

Resolved.
FYI for future ref: Was around 100% space in /var/
Wish Coldfusion would have errored and let us know this
though. Hopefully 8 will be better.

CF8 Verity "The collection you specified does not exist or is not registered with the ColdFusion Search Service."

I'm running ColdFusion 8 Enterprise on linux. I'm able to
create collections and index them through cfadmin as well as in cfm
application pages, but when trying to search I get the error
"The collection you specified does not exist or is not
registered with the ColdFusion Search Service."
I'm using the collection name in cfsearch and not the full
path.

Would I have been better off posting this in the General
Discussion section? Could the moderators move it if so,
please?

CF8 Verity Collection Problem

I am running the test code shown below.
The result is:
query
CATEGORIES CHARSET CREATED DOCCOUNT EXTERNAL LANGUAGE
LASTMODIFIED MAPPED NAME ONLINE PATH REGISTERED SIZE
1 No 1252 {ts '2008-12-11 15:51:16'} 81 NO english {ts
'2008-12-11 15:52:05'} NO testcontent YES
/opt/coldfusion8/verity/collections/testcontent CF 220
struct
Detail The collection you specified does not exists or is not
registered with the ColdFusion Search Service.
Message The collection testcontent does not exist.
[snip rest of dump]
When I use the CLI rcvdk program, the collection works fine
and I can search against the collection.

Dessimis wrote:
> I am running the test code shown below.
>
> The result is:
>
> query
> CATEGORIES CHARSET CREATED DOCCOUNT EXTERNAL LANGUAGE
LASTMODIFIED
> MAPPED NAME ONLINE PATH REGISTERED SIZE
> 1 No 1252 {ts '2008-12-11 15:51:16'} 81 NO english {ts
'2008-12-11
> 15:52:05'} NO testcontent YES
> /opt/coldfusion8/verity/collections/testcontent CF 220
>
> struct
> Detail The collection you specified does not exists or
is not registered with
> the ColdFusion Search Service.
> Message The collection testcontent does not exist.
> [snip rest of dump]
>
> When I use the CLI rcvdk program, the collection works
fine and I can search
> against the collection.
>
> <cfcollection action = "list" name = "verity" >
>
> <cfoutput>List of all known verity
collections.<br><br></cfoutput>
> <CFDUMP var = "#verity#">
> <br>
> <cftry>
> <cfsearch name = "quick"
> collection="southernrefecontent"
You're searching the "southernrefecontent" collection here,
not
testcontent (unless you anonymized the name before sending).
Mack

Verity Indexing Error

I am receiving this error and not sure what to do to fix
this. I am able to create a collection but I can't add the index.
The cfindex code I have is creating an index from a query of data.
I know this all works because my development machine works fine.
When I upload to the host this breaks. Here is the error:

No that directory did not exist and was not being created
under CFMX 7. Under 7 it works fine. When I move things over to a
production MX 6 server it gives me that error. I created that
custom folder and that error did stop, however it isn't actually
indexing anything now and also not giving me any errors. I'm
running test cfsearches against the collection, dumping the results
and nothing is showing. Any idea why it would not index anything?
Again, this all works fine under MX7...

Verity categoryTree Bug?

I have a verity index that has categories enabled. I am using
both category and categoryTree to narrow the search. I am using
search type="internet".
This issue does not occur in CF8, but is occurring in CF7.
Correct Results:
<cfsearch name="searchResults"
collection="collection_name"
category=""
categoryTree=""
type="internet"
criteria="search test">
This search returns everything as expected.
Incorrect Results:
<cfsearch name="searchResults"
collection="collection_name"
category=""
categoryTree="A"
type="internet"
criteria="search test">
This search finds items that match the criteria just fine,
but the resulting #searchResults# has manipulated data for the
returned category value. Additionally, it is finding products that
don't even have a categoryTree="A" and showing that it is a
categoryTree="A".
This is a very odd issue and hopefully I've explained it
enough for someone to understand. Let me know if you need more
details. Has anyone experienced anything like this? Eventually
we'll be upgrading our servers to CF8, but in the meantime, I'd
like to see if there is a work-around to get my search results
accurate.
Thanks

I have a verity index that has categories enabled. I am using
both category and categoryTree to narrow the search. I am using
search type="internet".
This issue does not occur in CF8, but is occurring in CF7.
Correct Results:
<cfsearch name="searchResults"
collection="collection_name"
category=""
categoryTree=""
type="internet"
criteria="search test">
This search returns everything as expected.
Incorrect Results:
<cfsearch name="searchResults"
collection="collection_name"
category=""
categoryTree="A"
type="internet"
criteria="search test">
This search finds items that match the criteria just fine,
but the resulting #searchResults# has manipulated data for the
returned category value. Additionally, it is finding products that
don't even have a categoryTree="A" and showing that it is a
categoryTree="A".
This is a very odd issue and hopefully I've explained it
enough for someone to understand. Let me know if you need more
details. Has anyone experienced anything like this? Eventually
we'll be upgrading our servers to CF8, but in the meantime, I'd
like to see if there is a work-around to get my search results
accurate.
Thanks

CF8 Verity Indexing

Similar Messages

Maybe you are looking for