Major crawler issues - removal of trailing slashes from URLs and (seemingly) poor regex implementation

Question

I've been suffering from two issues when setting up a crawl of an intranet website hosted on a proprietary web CMS using SharePoint 2010 Search. The issues are:
1: The crawler appears to remove trailing slashes from the end of URLs.
In other words, it turns http://mysite.local/path/to/page/ into http://mysite.local/path/to/page. The CMS can cope with this as it automatically forwards requests for http://mysite.local/path/to/page to http://mysite.local/path/to/page/ -
but every time it does this it generates a warning in the crawl - one of the "URL was permanently moved" warnings. The URL hasn't been moved, though, the crawler has just failed to record it properly. I've seen a few posts about this in various places,
all of which seem to end without resolution, which is particularly disappointing given that URLs in this format (with a trailing slash) are pretty common. (Microsoft's own main website has a few in links on its homepage - http://www.microsoft.com/en-gb/mobile/
for instance).
The upshot of this is that a crawl of the site, which has about 50,000 pages, is generating upwards of 250,000 warnings in the index, all apparently for no reason other than the crawler changes the address of all the pages it crawls? Is there a fix for this?
2: The Regex implementation for crawl rules does not allow you to escape a question mark
... despite what it says
here: I've tried and tested escaping a question mark in a url pattern by surrounding it in square brackets in the regex, i.e.: [?] but regardless of doing so, any URL with a question mark in just falls right through the rule. As soon as you remove the 'escaped'
(i.e. seemingly not actually escaped at all) question mark from the rule, and from the test URL pattern, the rule catches, so it'd definitely not escaping it properly using the square brackets. The more typical regex escape pattern (a backslash before the
character in question) doesn't seem to work, either. Plus neither the official documentation on regex patterns I've been able to find, nor the book I've got about SP2010 search, mention escaping characters in SP2010 crawl rule regexes at all. Could it be that
MS have released a regex implementation for matching URL patterns that doesn't account for the fact that ? is a special character in both regex and in URLs?
Now I could just be missing something obvious and would be delighted to be made to look stupid by someone giving me an easy answer that I've missed, but after banging my head against this for a couple of days, I really am coming to the conclusion that Microsoft
have released a regex implementation for a crawler that doesn't work with URL patterns that contain a question mark. If this is indeed the case, then that's pretty bad, isn't it? And we ought to expect a patch of some sort? I believe MS are supporting SP2010
until 2020? (I'd imagine these issues are fixed in 2013 Search, but my client won't be upgrading to that for at least another year or two, if at all).
Both these issues mean that the crawl of my client's website is taking much longer, and generating much more data, than necessary. (I haven't actually managed to get to the end of a full crawl yet because of it... I left one running overnight and I just
hope it has got to the end. Plus the size of the crawl db was 11GB and counting, most of that data being pointless warnings that the crawler appeared to be generating itself because it wasn't recording URLs properly). This generation of pointless mess
is also going to make the index MUCH harder to maintain. 
I'm more familiar with maintaining crawls in Google systems which have much better regex implementations - indeed (again, after two days of banging my head against this) I'd almost think that the regex implementation in 2010 search crawl rules was cobbled
together at the last minute just because the Google Search Appliance has one. If so (and if I genuinely haven't missed something obvious - which I really hope I have) I'd say it wasn't worth Microsoft bothering because the one they have released appears to
be entirely unfit for purpose?
I'm sure I'm not the first person to struggle with these problems and I hope there's an answer out there somewhere (that isn't just "upgrade to 2013")?
I should also point out that my client is an organisation with over 3000 staff, so heaven knows how much they are paying MS for their Enterprise Agreement. Plus I pay MS over a grand a year in MSDN sub fees etc, so (unless I'm just being a numpty) I would
expect a higher standard of product than the one I'm having to wrestle with at the moment.

Answer 1

Hi David,
as i know in sharepoint 2010 crawl there is a rule to include or exclude the URL that using '?' characters, if i may know have you implemented the rule?
In the Crawl Configuration section, select one of the following options:
Exclude all items in this path. Select this option if you want to exclude all items in the specified path from crawls. If you select this option, you can refine the exclusion by selecting the following:
Exclude complex URLs (URLs that contain question marks (?)). Select this option if you want to exclude URLs that contain parameters that use the question mark (?) notation.
Include all items in this path. Select this option if you want all items in the path to be crawled. If you select this option, you can further refine the inclusion by selecting any combination of the following:
Follow links on the URL without crawling the URL itself. Select this option if you want to crawl links contained within the URL, but not the starting URL itself.
Crawl complex URLs (URLs that contain a question mark (?)). Select this option if you want to crawl URLs that contain parameters that use the question mark (?) notation.
Crawl SharePoint content as HTTP pages. Normally, SharePoint sites are crawled by using a special protocol. Select this option if you want SharePoint sites to be crawled as HTTP pages instead. When the content is crawled by using the HTTP
protocol, item permissions are not stored.
for the trailing slash issue, may i have your latest cumulative update or your sharepoint database number? as i remember there was a fix on SP1 + june update regarding trailing slash, but not quite sure if the symptoms are the same with your environment.
Sharepoint may use sharepoint connector regarding regex, but older regex may not able to have a capability to filter out the parameters so, a modification regarding the trailing slash may happend.
please let us know your feedback.
Regards,
Aries
Microsoft Online Community Support
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.