Obtaining a rowcount from a csv file before bulk insert

Question

I have a group of csv files that i am bulk inserting into my sql table.  They are varying lengths, but the last seven lines of every file are not lines of data that comply with the format of my table and are causing insert errors.  I know i can
specify a "lastrow" value for the bulk insert, but my problem is that i need a rowcount before running the statement and I can't seem to figure out a way of getting it with how im accessing the file.  (example below of what i'm doing without
checking for the qty of rows):
CREATE TABLE ALLFILENAMES(WHICHPATH VARCHAR(255),WHICHFILE varchar(255))
    --some variables
    declare @filename varchar(255),
            @path     varchar(255),
            @sql      varchar(8000),
            @cmd      varchar(1000),
@lastrow  integer
    --get the list of files to process:
SET @path = '\\chewbacca\PlantFloorData\RLData\'
    SET @cmd = 'dir ' + @path + '*.csv /b'
print @path
print @cmd
    INSERT INTO  ALLFILENAMES(WHICHFILE)
    EXEC Master..xp_cmdShell @cmd
    UPDATE ALLFILENAMES SET WHICHPATH = @path where WHICHPATH is null
    --cursor loop
    declare c1 cursor for SELECT WHICHPATH,WHICHFILE FROM ALLFILENAMES where WHICHFILE like '%.csv%'
    open c1
    fetch next from c1 into @path,@filename
    While @@fetch_status <> -1
      begin
  set @lastrow = 3 -- for now just to make sure logic below works
  print @lastrow
 --bulk insert won't take a variable name, so make a sql and execute it instead:
       set @sql = 'BULK INSERT HMIDATA.dbo.RawData FROM ''' + @path + @filename + ''' '
           + '     WITH ( 
                   FIELDTERMINATOR = '','', 
                   ROWTERMINATOR = ''\n'', 
                   FIRSTROW = 2 ,
   LASTROW = ' + STR(@lastrow) + '
    print @sql
    exec (@sql)
      fetch next from c1 into @path,@filename
      end
    close c1
    deallocate c1
    --Extras
    --delete from ALLFILENAMES where WHICHFILE is NULL
    --select * from ALLFILENAMES
    drop table ALLFILENAMES

Answer 1

You could do a SELECT COUNT(*) FROM OPENROWSET(BULK), but you would need a format file that specifies a single-field record format with \r\n as the terminator. With the COUNT(*) you could determine the correct value for LASTROW.
You could also load the entire file into a two-column table where one column is an IDENTITY column and the other is the raw data from the line in the file. And then you could crack the good rows.
...or you could write a client-side program or a SSIS package for the task.
Erland Sommarskog, SQL Server MVP, [email protected]