Unable to Enforce Unique Values, Duplicate Values Exist

Question

I have list in SP 2010, it contains roughly 1000 items.  I would like to enforce unique values on the title field.  I started by cleaning up the list, ensuring that all items already had a unique value.  To help with this, I used the export
to excel action, then highlight duplicates within Excel.  So as far as I can tell, there are no duplicates within that list column.
However, when I try to enable the option to Enforce Unique Values, I receive the error that duplicate values exist within the field and must be removed.
Steps I've taken so far to identify / resolve duplicate values:
- Multiple exports to Excel from an unfiltered list view, then using highlight duplicates feature > no duplicates found
- deleted ALL versions of every item from the list (except current), ensured they were completely removed by deleting from both site and site collection recycle bins
- Using the SP Powershell console, grabbed all list items and exported all of the "Title" type fields (Item object Title, LinkTitle, LinkTitleNoMenu, etc) to a csv and ran that through excel duplicate checking as well. 
Unless there's some rediculous hidden field value that MS expects anyone capable of attempting to enforce unique values on a list (which is simple enough for anyone to figure out - if it doesn't throw an error), then I've exhausted anything I can think
of that might cause the list to report duplicate values for that field.
While I wait to see if someone else has an idea, I'm also going to see what happens if I wipe the Crawl Index and start it from scratch.
- Jon

Answer 1

First, I create index for a column in list settings, it works fine no matter duplicate value exists or not;
then I set enforce unique values in the field, after click OK, I get duplicate values error message.
With SQL Server profiler, I find the call to proc_CheckIfExistingFieldHasDuplicateValues and the parameters. After reviewing this stored procedure in content database,
I create the following script in SQL Server management studio:
declare @siteid
uniqueidentifier
declare @webid
uniqueidentifier
declare @listid
uniqueidentifier
declare @fieldid
uniqueidentifier
set @siteid='F7C40DC9-E5D3-42D7-BE60-09B94FD67BEF'
set @webid='17F02240-CE04-4487-B961-0482B30DDA84'
set @listid='B349AF8D-7238-419D-B6C4-D88194A57EA7'
set @fieldid='195A78AC-FC52-4212-A72B-D03144DC1E24'
SELECT
* FROM TVF_UserData_List(@ListId)
AS U1 INNER
MERGE JOIN
            NameValuePair_Latin1_General_CI_AS
AS NVP1 WITH (INDEX=NameValuePair_Latin1_General_CI_AS_MatchUserData)
ON NVP1.ListId
= @ListId AND NVP1.ItemId
= U1.tp_Id
AND ((NVP1.Level
= 1 AND U1.tp_DraftOwnerId
IS NULL)
OR NVP1.Level
= 2)
AND NOT((DATALENGTH(ISNULL(NVP1.Value,
= 0)) AND U1.tp_Level
= NVP1.Level
AND U1.tp_IsCurrentVersion
= CONVERT(bit, 1)
AND U1.tp_CalculatedVersion
= 0 AND U1.tp_RowOrdinal
= 0 INNER
MERGE JOIN
            NameValuePair_Latin1_General_CI_AS
AS NVP2 WITH (INDEX=NameValuePair_Latin1_General_CI_AS_CI)
ON NVP2.SiteId
= @SiteId AND NVP2.ListId
= @ListId AND NVP2.FieldId
= @FieldId AND NVP2.Value
= NVP1.Value
AND NVP2.ItemId <> NVP1.ItemId
CROSS APPLY TVF_UserData_ListItemLevelRow(NVP2.ListId, NVP2.ItemId,
NVP2.Level, 0)
AS U2 WHERE ((NVP2.Level
= 1 AND U2.tp_DraftOwnerId
IS NULL)
OR NVP2.Level
= 2)
AND NOT((DATALENGTH(ISNULL(NVP2.Value,
= 0))      
I can find the duplicate list items based on the result returned by the query above.
Note that you need to change the parameter values accordingly, and change the name of NameValuePair_Latin1_General1_CI_AS table based on the last parameter of the
proc_CheckIfExistingFieldHasDuplicateValues stored procedure. You can review the code of this stored procedure by yourself.
Note that direct operation on the content database in production environment is not supported, please do all these in test environment.