CAS - Write text of certain html div tag to new attribute

Question

Hi,
Is it possible with CAS to extract text from certain HTML div tags to new attributes. So let's say we have tags in a specific "Tags" div and store that text in a new attribute of that record?
Thanks!
Marco Snels

Answer 1

Hey Marco,
I'd need to try this out a bit more since I have never built a custom CAS web crawler parser plugin. Here's what I know based on your questions:
1) If I understand it correctly the name after the METATAG_PROPERTY_NAME_PREFIX will be the attribute name (?) so in this case Endeca.Document.HTML.MetaTag (?)
Correct. You will want to change this in the code to something more meaningful to what you're trying to parse out of the <DIV> you're interested in.
2) And I think that I have to change the FILTER-HTMLMETATAG name after the getParseMeta().add line but I don't know to which value?
Accorinding to the WebCrawler guide this sample attribute was only added so you could see a new property on each output record to prove that this custom plugin was being executed. It can be removed for your custom plugin. That said, you're going to have to change what you're pulling out out the parser object. In the sample, they're pulling out the MetaTags (via an Endeca extension of the nutch parser), which you don't care about for your need. (e.g. HTMLMetaTags tags = parse.getData().getMetaTag();). Instead, you'll need to pull out the Content (e.g. parser.getData().getContentMeta()). You may want to google for nutch parser usage to get examples on how specific HTML elements can be found.
HTH,
Dan
http://branchbird.com