Exporting table data into text files

Question

I got a request to export the data from about 85 tables into 85 text files. I assume that they will want a header row and some delimiter character. This is a process that must run every night by a sql job.
Obviously I want it to be flexible. Tables could be addede and removed and columns could be added and removed. There will probably be a control table that has the list of tables to export.
Looked at bcp - but seems that it is command line only?
SSIS package?
What other options are there?

Answer 1

Wanted to post my solution. Ended up using bcp in a PowerShell Script. I was unable to use bcp in SQL because it requires xp_cmdshell and that is not allowed at many of our client sites. So wrote a Powershell script that executes bcp. The ps script
loops through a table that contains a row for each table/view to export along with many values and switches for the table/view export: export path, include column headers, enclose each field in double quotes, double up embedded double quotes, how to format
dates, what to return for NULL integers - basically the stuff that is not flexible in bcp. To get this flexibility I created a SQL proc that takes these values as input and creates a view. Then the PS bcp does a SELECT on the view. Many of my tables are
very wide and bcp has a limit of 4000/8000. Some of my SELECT statements ended up being GT 35k in length. So using a view got around this size limitation.
Anyway, the view creation and bcp download is really fast. It can download about 4 gb of data in 20 minutes. It can download it faster the 7z can zip it.
Below is the SQL proc to format the SELECT statement to create the view for bcp (or some other utility like SQLCMD, Invoke-SQLCMD) or SQL query by "SELECT * from v_ExportData". the proc can be used from SQL or PS, or anything that can call a SQL
Proc and then read a SQL view.
IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[ExportTablesCreateView]') AND type in (N'P', N'PC'))
DROP PROCEDURE [dbo].[ExportTablesCreateView]
GO
CREATE PROCEDURE [dbo].[ExportTablesCreateView]
ExportTablesCreateView
Description:
Read in a tablename or viewname with parameters and create a view v_ExportTable of the
data. The view will typically be read by bcp to download SQL table data to flat files.
bcp does not have the options to include column headers, include fields in double quotes,
format dates or use '0' for integer NULLS. Also, bcp has a limit of varhar 4000 and
wider tables could not be directly called by bcp. So create the v_ExportTable and have
bcp SELECT v_ExportTable instead of the original table or view.
Parameters:
@pTableName VARCHAR(128) - table or view to create v_ExportTable from
@pColumnHeader INT =1 - include column headers in the first row
@pDoubleQuoteFields INT = 1 - put double quotes " around all column values including column headers
@pDouble_EmbeddedDoubleQuotes INT = 1 - This is usually used with @pDoubleQuoteFields INT = 1. 'ab"c"d would be 'ab""c""d.
@pNumNULLValue VARCHAR(1) = '0' - NULL number data types will export this value instead of bcp default of ''
@pDateTimeFormat INT = 121 - DateTime data types will use this format value
Example:
EXEC ExportTablesCreateView 'custname', 1, 1, 1, '0', 121
@pTableName VARCHAR(128),
@pColumnHeader INT = 1,
@pDoubleQuoteFields INT = 1,
@pDouble_EmbeddedDoubleQuotes INT = 1,
@pNumNULLValue VARCHAR(1) = '0',
@pDateTimeFormat INT = 121
AS
BEGIN
DECLARE @columnname varchar(128)
DECLARE @columnsize int
DECLARE @data_type varchar(128)
DECLARE @HeaderRow nvarchar(max)
DECLARE @ColumnSelect nvarchar(max)
DECLARE @SQLSelect nvarchar(max)
DECLARE @SQLCommand nvarchar(max)
DECLARE @ReturnCode INT
DECLARE @Note VARCHAR(500)
DECLARE db_cursor CURSOR FOR
SELECT COLUMN_NAME, ISNULL(Character_maximum_length,0), Data_type
FROM [INFORMATION_SCHEMA].[COLUMNS]
WHERE TABLE_NAME = @pTableName AND TABLE_SCHEMA='dbo'
OPEN db_cursor
FETCH NEXT FROM db_cursor INTO @ColumnName, @ColumnSize, @Data_type
SET @HeaderRow = ''
SET @ColumnSelect = ''
-- Loop through each of the @pTableColumns to build the SELECT Statement
WHILE @@FETCH_STATUS = 0
BEGIN
BEGIN TRY
-- Put double quotes around each field - example "MARIA","SHARAPOVA"
IF @pDoubleQuoteFields = 1
BEGIN
-- Include column headers in the first row - example "FirstName","LastName"
IF @pColumnHeader = 1
SET @HeaderRow = @HeaderRow + '''"' + @ColumnName + '"'' as ''' + @columnname + ''','
-- Unsupported Export data type returns "" - example "",
IF @Data_Type in ('image', 'varbinary', 'binary', 'timestamp', 'cursor', 'hierarchyid', 'sql_variant', 'xml', 'table', 'spatial Types')
SET @ColumnSelect = @ColumnSelect + '''""'' as [' + @ColumnName + '],'
-- Format DateTime data types according to input parameter
ELSE IF @Data_Type in ('datetime', 'smalldatetime', 'datetime2', 'date', 'datetimeoffset')
-- example - CASE when [aaa] IS NULL THEN '""' ELSE QUOTENAME(CONVERT(VARCHAR,[aaa], 121), CHAR(34)) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN ''""'' ELSE QUOTENAME(CONVERT(VARCHAR,[' + @columnname + '],' + CONVERT(VARCHAR,@pDateTimeFormat) + '), CHAR(34)) END AS [' + @ColumnName + '],'
-- SET Numeric data types with NULL value according to input parameter
ELSE IF @Data_Type in ('bigint', 'numeric', 'bit', 'smallint', 'decimal', 'smallmoney', 'int', 'tinyint', 'money', 'float', 'real')
-- example - CASE when [aaa] IS NULL THEN '"0"' ELSE QUOTENAME([aaa], CHAR(34)) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN ''"' + @pNumNULLValue + '"'' ELSE QUOTENAME([' + @columnname + '], CHAR(34)) END AS [' + @ColumnName + '],'
ELSE
-- Double embedded double quotes - example "abc"d"ed" to "abc""d""ed". Only applicible for character data types.
IF @pDouble_EmbeddedDoubleQuotes = 1
BEGIN
-- example - CASE when [aaa] IS NULL THEN '""' ELSE '"' + REPLACE([aaa],'"','""') + '"' END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @ColumnName + '] IS NULL THEN ''""'' ELSE ''"'' + REPLACE([' + @ColumnName + '],''"'',''""'') + ''"'' END AS [' + @ColumnName + '],'
END
-- DO NOT PUT Double embedded double quotes - example "abc"d"ed" unchanged to "abc"d"ed"
ELSE
BEGIN
-- example - CASE when [aaa] IS NULL THEN '""' ELSE '"' + [aaa] + '"' END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @ColumnName + '] IS NULL THEN ''""'' ELSE ''"'' + [' + @ColumnName + '] + ''"'' END AS [' + @ColumnName + '],'
END
END
-- DO NOT PUT double quotes around each field - example MARIA,SHARAPOVA
ELSE
BEGIN
-- Include column headers in the first row - example "FirstName","LastName"
IF @pColumnHeader = 1
SET @HeaderRow = @HeaderRow + '''' + @ColumnName + ''' as ''' + @columnname + ''','
-- Unsupported Export data type returns '' - example '',
IF @Data_Type in ('image', 'varbinary', 'binary', 'timestamp', 'cursor', 'hierarchyid', 'sql_variant', 'xml', 'table', 'spatial Types')
SET @ColumnSelect = @ColumnSelect + ''''' as [' + @ColumnName + '],'
-- Format DateTime data types according to input parameter
ELSE IF @Data_Type in ('datetime', 'smalldatetime', 'datetime2','date', 'datetimeoffset')
-- example - CASE when [aaa] IS NULL THEN '''' ELSE CONVERT(VARCHAR,[aaa], 121) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN '''' ELSE CONVERT(VARCHAR,[' + @columnname + '],' + CONVERT(VARCHAR,@pDateTimeFormat) + ') END AS [' + @ColumnName + '],'
-- SET Numeric data types with NULL value according to input parameter
ELSE IF @Data_Type in ('bigint', 'numeric', 'bit', 'smallint', 'decimal', 'smallmoney', 'int', 'tinyint', 'money', 'float', 'real')
-- example - CASE when [aaa] IS NULL THEN '"0"' ELSE CONVERT(VARCHAR, [aaa]) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN ''' + @pNumNULLValue + ''' ELSE CONVERT(VARCHAR,[' + @columnname + ']) END AS [' + @ColumnName + '],'
ELSE
BEGIN
-- Double embedded double quotes - example "abc"d"ed" to "abc""d""ed". Only applicible for character data types.
IF @pDouble_EmbeddedDoubleQuotes = 1
-- example - CASE when [aaa] IS NULL THEN '' ELSE CONVERT(VARCHAR,REPLACE([aaa],'"','""')) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN '''' ELSE CONVERT(VARCHAR,REPLACE([' + @columnname + '],''"'',''""'')) END AS [' + @ColumnName + '],'
ELSE
-- example - CASE when [aaa] IS NULL THEN '' ELSE CONVERT(VARCHAR,[aaa]) END AS [aaa],
SET @ColumnSelect = @ColumnSelect + 'CASE WHEN [' + @columnname + '] IS NULL THEN '''' ELSE CONVERT(VARCHAR,[' + @columnname + ']) END AS [' + @ColumnName + '],'
END
END
FETCH NEXT FROM db_cursor INTO @ColumnName, @ColumnSize, @Data_Type
END TRY
BEGIN CATCH
RETURN (1)
END CATCH
END
CLOSE db_cursor
DEALLOCATE db_cursor
BEGIN TRY
-- remove last comma
IF @pColumnHeader = 1
SET @HeaderRow = SUBSTRING(@HeaderRow , 1, LEN(@HeaderRow ) - 1)
SET @ColumnSelect = SUBSTRING(@ColumnSelect, 1, LEN(@ColumnSelect) - 1)
-- Put on the finishing touches on the SELECT
IF @pColumnHeader = 1
SET @SQLSelect = 'SELECT ' + @HeaderRow + ' UNION ALL ' +
'SELECT ' + @ColumnSelect + ' FROM [' + @pTableName + ']'
ELSE
SET @SQLSelect = 'SELECT ' + @ColumnSelect + ' FROM [' + @pTableName + ']'
---- diagnostics
---- PRINT truncates at 4k or 8k, not sure, my tables have many colummns
--PRINT @SQLSelect
--DECLARE @END varchar(max) = RIGHT(@SQLSelect, 3000)
--PRINT @end
--EXECUTE sp_executesql @SQLSelect
-- drop view if exists -- using view because some tables are very wide. one of my tables had a 33k select statement
SET @SQLCommand = '
IF EXISTS (SELECT * FROM SYS.views WHERE name = ''v_ExportTable'')
BEGIN
DROP VIEW v_ExportTable
END'
EXECUTE @ReturnCode = sp_executesql @SQLCommand
IF @returncode = 1
BEGIN
RETURN (1)
END
-- create the view
SET @SQLCommand = '
CREATE VIEW v_ExportTable AS ' + @SQLSelect
-- diagnostics
--print @sqlcommand
EXECUTE @ReturnCode = sp_executesql @SQLCommand
IF @returncode = 1
BEGIN
RETURN (1)
END
END TRY
BEGIN CATCH
RETURN (1)
END CATCH
RETURN (0)
END -- CREATE PROCEDURE [dbo].[ExportTablesCreateView]
GO