Getting correct field length when creating formats for delimited flat files

Question

Using Data Services 3.2 to pull in tsv and Excel source files.
When creating a new format for each file, DS tries to intelligently set the field size for each column. However, it seems to sample only a portion of the file. The end result is that it often determines too short a length for a field.
Is there a way to force it to scan an entire file before it sets the field length? Or alternatively, how to get it to at least throw an error instead of silently truncating my data?

Answer 1

I am not aware of any way to force DS to scan the whole file, or to log a warning when it truncates.  The field sizes DS sets by using a sample of the data is only really to give you a starting point, you need to try and determine the maximum possible lengths preferably from someone who understands the source of the data or through profiling.  But the profiling method will obviously only be based on the data you have at the time of profiling, new data added to the source later could exceed the field lengths.  The way I prefer to handle this is to set all the field lengths to longer than expected, eg. varchar(999) and then use a validation transform immediately after reading the file to check for any fields that exceed the maximum allowed for the target system and send these to an error file/table.