How to access Properties in Custom Crawler

Question

[Plumtree 5.0.2 / .NET]
Hi,
Where in the source code for the NT File Crawler does the accessor (in this case the html accessor) pick up the metadata from the document and input the values into the Plumtree Cards?
To explain - we are developing a custom crawler that will crawl html formatted files containing metadata about the documents to be indexed, and not crawling the original documents themselves. (I can explain the reasons for this in more detail if relevant)
One of the values included in this metadata is the location of the original file which I need the crawler to make the ClickThroughURL.
Rather than parsing the html document myself, I figured the HTML Accessor is already doing the job for me. The field is mapped as a property in the document type for this crawler and I just need to work out how to to access that to replace:
data.ClickThroughURL = CurrentFile.FullName;
with:
data.ClickThroughURL = MY_VALUE_FROM_PROPERTIES;
I'm struggling a bit through lack of doco but I thought if I could find the section of the code where the crawler sets the properties for each document I'd be able to see how I can access the collection - I'd thought it would be in IDocument.GetMetaData, but I can't spot it.
I hope this makes sense to someone out there....
thanksAni

Answer 1

Hi Clinton,
I checked PTSpy and it appears to be looking at the right URL, but receiving a 404. PTSpy records the following, (I've changed the server name but it was correct):
Entering Function CPTUserCookieDatabase::AddHeadersToCookies: parameters: lpctstrDomain = APP_SERVER_WHERE_CRAWLER_LOCATED.hosting.companyname.net; lpctstrObject = /ANITA/DJCustomCrawlerWS/XUIService.asmx; lpctstrHeaders = HTTP/1.1 404 Not FoundServer: Microsoft-IIS/5.0Date: Wed, 02 Mar 2005 03:12:54 GMTX-Powered-By: ASP.NETX-AspNet-Version: 1.1.4322Cache-Control: privateContent-Type: text/html; charset=utf-8Content-Length: 1530; lCookieSegment = -1;
I wondered if it's trying to look for "/ANITA/DJCustomCrawlerWS/XUIService.asmx" in the portal server or whether it's looking in the lpctstrDomain where the file actually exists. Would you expect to see a fully qualified URL for lpctstrObject? Either way, as I said there is no 404s recorded in the IIS logs on either servers.
==================================
Back to the question of getting a customized property value from the Crawler. Can you explain a little more about "CrawlerConstants.TAG_PROPERTIES" and what I'm looking for please? I can see that this is getting its value from the web.config key "UseRemoteProperties" which in my instance had the value of "0" i.e. USE_LOCAL.
The custom property I want to get the value of is called "FileLocation" and when i go to the knowledge directory and check the properties page of the documents I'm crawling the value is right there in the "Customized Properties" exactly as expected whether I've got "UseRemoteProperties" set to 0 or 1 (and it's refreshed any time I change it and re-run the crawl).
In Document.cs (GetMetaData()) the DocumentMetaData() object is instantiated and the method Put() is used to populate some of the document's property values, and there is the comment that they are overwritten by the accessor depending on the "proper setting in the web.config file" which is the CrawlerConstants.TAG_PROPERTIES that you spoke of.
So where does the accessor do this? Somewhere, something (the accessor presumably) is writing all the other custom properties to the card. I've tried to use the Get() method for DocumentMetaData:
data.Get("FileLocation");
to retrieve the value for the one I'm after but when I write this out to the log it's always blank, the value hasn't been set yet. Where and how can I retrieve this value to update the clickThroughURL with it?
Thank you again. Please shout if I haven't explained this very well, I'm wondering whether my lack of understanding is making it hard for me to phrase my questions, but I'm getting desperate, I've spent far to much time and money on this.
Ani