What is the source of this Crawl Error `The item could not be indexed successfully because the item failed in the indexing subsystem`

Question

Once in a while my full-crawl fails and stops working properly. The crawllogs show this error for all items crawled.
The item could not be indexed successfully because the item failed in the indexing subsystem. ( The item could not be indexed successfully because the item failed in the indexing subsystem.; Caught exception when preparing generation GID[7570]: (Previous generation
(last=GID[7569], curr=GID[7570]) is still active. Cannot open GID[7570]); Aborting insert of item in Link Database because it was not inserted to the Search Index.; ; SearchID = F201681E-AF1B-45D2-BFFD-6A2582D10C19 )
The full crawls starts out ok, after a while (1.5 hours into the process, 50% of all the data) suddenly no more items can be added to the index.
The index seems to be stuck. The index files on disk are no longer updated (Located in D:\Microsoft
Office Servers\15.0\Data\Office Server\Applications\Search\Nodes\BAADC4\IndexComponent3\storage\data\SP4d91e6081ac3.3.I.0.0\ms\%default) The Index and Admin component start to report these error in the ULS logs:
NodeRunnerIndex: Journal[SP4d91e6081ac3]:
Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
NodeRunnerIndex: Remote
service invocation: method=RollbackGeneration() Service = {  Implementation type=Microsoft.Ceres.SearchCore.ContentTargets.IndexRouter.IndexRouter  Component: SP4d91e6081ac3I.0.0.IndexRouter  Exposer Name: GenerationContentTarget} terminated
with exception: System.InvalidOperationException: Illegal state transition in SP4d91e6081ac3I.0.0.FastServer.FSIndex: Rollback -> Rollback
NodeRummerAdmin: RetryableInvocation[SP4d91e6081ac3]:
Exception invoking index cell I.0.0. Retrying in 16 seconds: System.InvalidOperationException: Illegal state transition in SP4d91e6081ac3I.0.0.FastServer.FSIndex: Rollback -> Rollback
Looks to me the index has troubles updating/merging 'generations'. But the exact working of the indexer is not documented (as far is I know). Let alone how to fix this.
Other (maybe related) observations 
Just before the errors start the NodeRunnerIndex starts a 'checkpoint': Journal[SP4d91e6081ac3]:
Starting checkpoint because forceCheckpoint is true.which ends a few moments later with Journal[SP4d91e6081ac3]:
All journal users have completed checkpoint Checkpoint[7560-7569].
Also just before the errors start to appear a TimerJob starts: Name=Timer
Job job-application-server-admin-service. This timerjobs does some strange things to the search topology: Synchronizing
Search Topology for application 'Search Service Application' with active topology [...] and Activating
components. Previous topology:   ---  New Topology: TopologyId: [...] followed by Starting
to execute Index RedistributeData method.
And right after these two evente the errors start to occur. (each row is a ULS log enrty)
INFO : fsplugin: IndexComponent3-bd83a8aa-923b-4526-97e8-47eac0986ff7-SP4d91e6081ac3.I.0.0 (4236): Prepare generation: 324 documents
IndexRouter[SP4d91e6081ac3]: Caught exception when preparing generation GID[7570]: (External component has thrown an exception.): System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
GenerationDispatcher[SP4d91e6081ac3]: Failed to prepare GID[7570] in 453 ms, failed on cells: [I.0.0], stale services: []
Journal[SP4d91e6081ac3]: Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
Remote service invocation: method=RollbackGeneration() Service = { Implementation type=Microsoft.Ceres.SearchCore.ContentTargets.IndexRouter.IndexRouter Component: SP4d91e6081ac3I.0.0.IndexRouter Exposer Name: GenerationContentTarget} terminated with exception: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
RetryableInvocation[SP4d91e6081ac3]: Exception invoking index cell I.0.0. Retrying in 2 seconds: System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
Journal[SP4d91e6081ac3]: Rolling back GID[7570] to GID[7569] prepGen=GID[7569]
The Question What is causing this? And how to prevent it? It happend twice in two weeks
nows. Out of the blue, no config change has been made, all disks have enough space.
Known Fix This resolves the problem, but doesn's address the root-cause of the problem!
Stop all crawls.
Wait a few minutes to let the crawl come to a complete stop.
Reset the index (clearing all!)
Start a full-crawl. In the meantime no search is available to the end user (boohoo!)

Answer 1

Hi,
I searched for the similar error log, the issue is finally solved by adding more drive space even though they though there is plenty space already.
https://social.technet.microsoft.com/Forums/office/en-US/d06c9b2c-0bc1-44c6-b83a-2dc0b70936c4/the-item-could-not-be-indexed-successfully-because-the-item-failed-in-the-indexing-subsystem?forum=sharepointsearch
http://community.spiceworks.com/topic/480369-the-item-could-not-be-indexed-successfully
From your decription, the issue seems to occur to your full crawl. There is one point in best practise for crawling:
Run full crawls only when it is necessary, the reasons to do a full crawl are as below:
https://technet.microsoft.com/en-us/library/jj219577.aspx#Plan_full_crawl
https://technet.microsoft.com/en-us/library/dn535606.aspx
Regards,
Rebecca Tu
TechNet Community Support
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact
[email protected]