.csv's, excel and ssis

Question

Hi we run 2012 std.  I've been using ssis now for 7 years or more and have always (especially lately) shuddered at the thought of needing to bake excel file functionality into my pkgs.  Some of the nightmares I remember include 32 bit directory
limitations, special add ons to the sql server, column typing, jet engines, compatibilities, leading zeros, sheets etc etc.  Maybe I'm wrong but i'm under the impression that full/custom control of a .csv's content involves creating it with excel.
My personal feeling is that it just shouldn't be that difficult.  But that it is and we never know what new surprise the next release of either product brings.
I'm faced with baking some .csv file creation into my pkg and wonder...can I use a more basic approach to creating these files?  One that uses flat file technology if I'm willing to do the formatting?  I vaguely remember testing this
theory (by putting my own delimeters in) once and the file did indeed open in excel.     
I wonder about metadata (related to column headings, data types etc) that might accompany the data in such a file when a .csv is created from excel instead of flat file approach.  I wonder about commas that r in the char data as opposed to being delimeters. 
Maybe I'll wish I'd never gone down this road because I'd lose control over managing such metadata if the need should arise.  My consumer is a separate business and I may not be in a position to tell them to live with the generic way the file
was created.
Does anybody feel my pain?  Any suggestions?  I'm also thinking about reusability thru dmv's, metadata in control tables etc.

Answer 1

here is the solution I am going with.  It is mostly untested and in need of review for efficiency and maintainability.  Its a first draft.
I am not a fan of ssis's flat file components.  They affect time to market in an agile environment.   I am a fan of reusability.  I haven't considered yet how this solution can work with linked servers.
Basically what I've tried to achieve here is dynamic functionality that can take a subset of any table's columns and format them into either a csv or fixed positional multiple row resultset (with a hdr if format is csv).  Both ID (unique
record identifier) and FileInstanceId (surrogate for original file name) must minimally be present on the source data table whose columns r being formatted for eventual landing in a flat file.  it would be ssis's responsibility to land the resultset
returned by this tool into a flat file.  
There r two metadata tables, one at the table level and one at the column level.  Constants r allowed in the 2nd.  There is a proc and a function.  The function made the proc slightly more readable.  There is an option for double quotes
around string values.  Presently, there r no constraints preventing bogus/contradictory data from being entered into the metadata tables.
USE [somedb]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[ourAppNameSourceDataTables](
[ID] [int] IDENTITY(1,1) NOT NULL,
[TableName] [varchar](128) NULL,
[DatabaseName] [varchar](128) NULL,
[Format] [varchar](25) NULL,
[IsActive] [bit] NULL,
[ModifiedDate] [datetime] NOT NULL,
CONSTRAINT [PK_ourAppNameSourceDataTables] PRIMARY KEY CLUSTERED
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[ourAppNameSourceDataTables] ADD DEFAULT (getdate()) FOR [ModifiedDate]
GO
USE [somedb]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[ourAppNameSourceDataColumns](
[ID] [int] IDENTITY(1,1) NOT NULL,
[FK_ourAppNameSourceDataTables] [int] NULL,
[TargetColumnName] [varchar](128) NULL,
[Order] [smallint] NULL,
[SourceColumnName] [varchar](128) NULL,
[IsConstant] [bit] NULL,
[ConstantValue] [varchar](100) NULL,
[FromColumn] [smallint] NULL,
[ToColumn] [smallint] NULL,
[AlsoDoubleQuoteDelimit] [bit] NULL,
[IsActive] [bit] NULL,
[ModifiedDate] [datetime] NOT NULL,
CONSTRAINT [PK_ourAppNameSourceDataColumns] PRIMARY KEY CLUSTERED
[ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [dbo].[ourAppNameSourceDataColumns] ADD DEFAULT (getdate()) FOR [ModifiedDate]
GO
USE [somedb]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[usp_createOurAppNameFiles]
@table varchar(128),
@database varchar(128),
@FileInstanceID bigint
AS
BEGIN
SET NOCOUNT ON;
BEGIN TRY
declare @format varchar(25)
declare @fileID int
declare @hdrVar varchar(max) = null
declare @lowRecordId bigint
declare @highRecordId bigint
declare @currentID bigint
declare @rowCount bigint
declare @dtlVar varchar(max)
declare @tblFinal table (id int primary key, record varchar(max)) -- holds final results
--get file properties
select @format = [Format],
@fileID = [ID]
from [dbo].[ourAppNameSourceDataTables]
where [IsActive] = 1 and
[TableName] = @table and
[DatabaseName] = @database
--build a header if csv, coalesce can also be used
if @format = 'csv' begin
select @hdrVar =
(SELECT
',' + TargetColumnName
FROM [dbo].[ourAppNameSourceDataColumns]
where [FK_ourAppNameSourceDataTables] = @fileID and
IsActive=1
order by [Order]
for xml path (''))
insert @tblFinal select -1, substring(@hdrVar,2, len(@hdrVar) -1)
end
--data type charactersitics
create table #Meta (SourceColumnName varchar(128) primary key,
Characteristics varchar(100))
declare @sqlMeta varchar(max) = 'insert #Meta select COLUMN_NAME, DATA_TYPE from ' +
@database +
'.[INFORMATION_SCHEMA].[COLUMNS] where TABLE_SCHEMA + ' +
'''' + '.' + '''' + ' + TABLE_NAME = ' + '''' + @table + ''''
exec (@sqlMeta)
--template
declare @tblTemplate table (ID int primary key, [Order] smallint, SourceColumnName varchar(128),
IsConstant bit, ConstantValue varchar(100),
FromColumn smallint, ToColumn smallint, AlsoDoubleQuoteDelimit bit, Characteristic varchar(100))
insert @tblTemplate
select a.*,Characteristics from
select ID,[Order],case when SourceColumnName is null then '' else SourceColumnName end SourceColumnName,
IsConstant,ConstantValue,FromColumn,ToColumn,AlsoDoubleQuoteDelimit
from [dbo].[ourAppNameSourceDataColumns]
where [IsActive] =1 and
[FK_ourAppNameSourceDataTables] = @fileId
) a
left join #Meta b
on a.SourceColumnName = b.SourceColumnName
drop table #Meta
--create the equivalent of an unpivot query for the data
declare @sqlPivotQuery varchar(max)
SELECT @sqlPivotQuery = (
select
' union all select ID,' +
case when IsConstant = 1 then '''' + case when ConstantValue is null then '' else ConstantValue end + ''''
else 'cast(case when ' + SourceColumnName + ' is null then ' + '''' + '''' + ' else ' + SourceColumnName + ' end as varchar(max))' end +
',' +
cast([ORDER] as varchar(5)) +
',' + '''' + case when characteristic is null then '' else characteristic end + '''' + ',' +
case when FromColumn is null then 'null' else cast(FromColumn as varchar(5)) end + ',' +
case when ToColumn is null then 'null' else cast(ToColumn as varchar(5)) end + ',' +
cast(AlsoDoubleQuoteDelimit as char(1)) +
' from ' +
@database + '.' + @table +
' where FileInstanceId = ' + cast(@FileInstanceId as varchar(19))
FROM @tblTemplate
order by [Order]
for xml path (''))
set @sqlPivotQuery = 'insert #Pivot ' + substring(@sqlPivotQuery,12,len(@sqlPivotQuery) - 9)
--unpivot the data
create table #Pivot (pk int identity(1,1) primary key, ID bigint,value varchar(max),[order] smallint, characteristic varchar(100),
FromColumn smallint,ToColumn smallint, AlsoDoubleQuoteDelimit bit)
create index ix on #Pivot(ID)
exec (@sqlPivotQuery)
--build required records
select @rowCount = count(*) from #Pivot
if @rowCount > 0
begin
select @lowRecordId = min(ID) from #Pivot
select @highRecordId = max(ID) from #Pivot
set @currentID = @lowRecordId
While @currentId <= @highRecordId
begin
set @dtlVar = ''
select @dtlVar =
(SELECT
case when @format = 'csv' then ',' else '' end +
case when AlsoDoubleQuoteDelimit = 1 then '"' else '' end +
dbo.udf_ourAppNameFieldFormat (value,characteristic,FromColumn, ToColumn) +
case when AlsoDoubleQuoteDelimit = 1 then '"' else '' end
FROM #Pivot
where [ID] = @currentId
order by [Order]
for xml path (''))
insert @tblFinal
select @currentId,case when @format = 'csv' then substring(@dtlVar,2,len(@dtlVar) -1) else @dtlvar end
select @currentId = min(ID) from #Pivot where ID > @currentId
if @currentId is null set @currentId = @highRecordId + 1
end
end
drop table #Pivot
select record from @tblFinal order by id
END TRY
BEGIN CATCH
DECLARE @ErrorMessage VARCHAR(500),@ErrorSeverity VARCHAR(10),@ErrorState CHAR(4)
SELECT
@ErrorMessage = ERROR_MESSAGE(),
@ErrorSeverity = ERROR_SEVERITY(),
@ErrorState = ERROR_STATE();
RAISERROR (@ErrorMessage, -- Message text.
@ErrorSeverity, -- Severity.
@ErrorState -- State.
END CATCH
END
GO
USE somedb
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE FUNCTION [dbo].[udf_ourAppNameFieldFormat]
@value varchar(max),
@characteristic varchar(100),
@FromColumn smallint,
@ToColumn smallint
RETURNS varchar(max)
AS
BEGIN
declare @return varchar(max) = ltrim(rtrim(case when @value is null then '' else @value end))
set @return = case when @characteristic like '%date%' then
case when @return = '' then @return
else CONVERT(VARCHAR(8), cast(@return as datetime), 112)
end
else @return
end
if @FromColumn is null or @ToColumn is null return @return
if @ToColumn < @FromColumn return null
if len(@return) >= @ToColumn - @FromColumn + 1 return substring(@return,1,@ToColumn - @FromColumn + 1)
set @return = @return + replicate(' ', @ToColumn - @FromColumn + 1 - len(@return))
RETURN @return
END
GO