Flat files data comma separated using SSIS.

Question

Hi,
I have multiple flat files which come in comma separated columns. See example below :
Customer Data
CustID,FName,LName,Disease,Email,Phone
12345,Xyz,Smit,Bronchitis, Asthma and fever,[email protected],80000000
12346,Abc,Doe,fever Headache,[email protected],90000000
12347,Klu,joe,Sugar, cough and fever,[email protected],12345678
Please look at the ID's 12345 and 12347. The disease column has a internal comma space between. How do i remove the comma spaces in the disease column, so that it can be loaded from flat file to sql table using SSIS. ?
Please help !
Thanks

Answer 1

Here is a full solution base on my post above (first option)
1. create temp table (Give it a unique name):
create table #T (Txt NVARCHAR(MAX))
GO
2. Insert all the data into temporary table. Each line in the text file, is a value for one column in a row in the table.
-- I will jump to the table and use simple insert.
-- If you have problem with step 1 then please inform us (this is simple bulk insert basically)
insert #T (Txt) values
('1234435,Xyz,Stemit,Brfsdonchitis, Asthma and fever,[email protected],80000000'),
('12346,Agjdfjbc,Doge,fevhhhher Headsxdshhache,[email protected],90000000'),
('123447,Klu,joe,Sugar, cough and fever,[email protected],12345678')
GO
the result should be like this:
Txt
1234435,Xyz,Stemit,Brfsdonchitis, Asthma and fever,[email protected],80000000
12346,Agjdfjbc,Doge,fevhhhher Headsxdshhache,[email protected],90000000
123447,Klu,joe,Sugar, cough and fever,[email protected],12345678
I use a SPLIT Function named Split_CLR_Fn. This is a CLR Split function that get input <string to split> and <string as delimiter,> and it return table with 2 columns ID, SplitData
For example if you use: SELECT * from Split_CLR_Fn('text1,text2,text3,',') then you get result:
ID SplitData
1 Text1
2 Text2
3 Text3
** You can find in the internet several good functions, I HIGHLY RECOMMENDED NOT TO USE T-SQL FUNCTIONS but CLR FUNCTION. Check thi link to understand why:
http://sqlperformance.com/2012/07/t-sql-queries/split-strings
** This is the best function that I know about and I use it, but I change the code a bit to return 2 columns and not just the SplitData as in this blog: http://sqlblog.com/blogs/adam_machanic/archive/2009/04/28/sqlclr-string-splitting-part-2-even-faster-even-more-scalable.aspx
That's it :-) we are ready for the solution which is very simple
Solution 1 (BAD solution but easy to write):
select
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 1) CustID,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 2) FName,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 3) LName,
STUFF((select ',' + SplitData from Split_CLR_Fn(Txt,',') where ID > 3 and ID < (select MAX(ID) from Split_CLR_Fn(Txt,',')) - 1 for XML path('')), 1 , 1,'') Disease,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = (select MAX(ID) from Split_CLR_Fn(Txt,',')) - 1) Email,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = (select MAX(ID) from Split_CLR_Fn(Txt,','))) Phone
from #T
GO
Solution 2: better in this case since the format is constant (this is the solution I wrote about above)
;With MyCTE as (
select
Txt,
SUBSTRING(Txt, 1, CHARINDEX(',', Txt, 1) - 1) as CustID
, SUBSTRING(
Txt
,CHARINDEX(',', Txt, 1) + 1 -- I start from the end of preview len
, CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)- CHARINDEX(',', Txt, 1) - 1
) as FName
, SUBSTRING(
Txt
,CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)+1 -- I start from the end of preview len
, CHARINDEX(',', Txt, CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)+1) - CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1) - 1
) as LName
, RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1) as Phone
, RIGHT(LEFT(Txt, Len(Txt) - Len(RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1)) - 1), CHARINDEX(',', REVERSE(LEFT(Txt, Len(Txt) - Len(RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1)) - 1)), 1) - 1) as Email
from #T
select CustID,FName,LName, Phone, Email, SUBSTRING(Txt, Len(CustID) + Len(FName) + Len(LName) + 4, Len(Txt) - Len(Email) - LEN(Phone) - Len(CustID) - Len(FName) - Len(LName) - 5) as Disease
from MyCTE
I hope that this is useful :-)
  Ronen Ariely
 [Personal Site]    [Blog]    [Facebook]