Deleting old flat files inside a SSIS package - SSIS 2005
Hi,
after each flat file import, I need to move the acquired file into an history folder and deleting old files of 1 week ago.
How can I cancel these files, acting inside my SSIS 2005 pckg?
Many thanks
While in foreach loop, after a successful import just move the file with the File System Task. Deleting old files is a little trickier, you can also loop thru the files and delete those older than based on name (if the date in the name of the file) or use
File Properties Task to get file's age to a variable and then use a precedence constraint to decide on deleting or not
conditionally.
Your question is general. Many devs did this before, just Bing/Google examples out.
Arthur My Blog
Similar Messages
-
I have looked on Ola's site (https://ola.hallengren.com/frequently-asked-questions.html) and I did see this:
DatabaseBackup is not deleting old backup files. What could the problem be?
Verify that the SQL Server and SQL Server Agent service accounts have full control of the backup directory.
If that directory is a network share, verify that the SQL Server and SQL Server Agent service accounts have full control of the network share.
If you are using a proxy account, verify that the account is a member of the sysadmin server role and that it has full control of the backup directory and network share.
Verify that the file is not locked in the file system; for example, a backup or antivirus software could be locking the file.
DatabaseBackup has been designed not to delete transaction log backups that are newer than the most recent full or differential backup. This could explain why transaction log backups are not being deleted.
I followed these instructions and then I right clicked the "sp_delete_backuphistory" job under "SQL Server Agent > Jobs" folder and clicked the "Start job at step ..." option. The job successfully ran, but my .bak backup
files were still on located on my separate drive partition I use specifically for these backups.
The SQL Server and the SQL Server Agent are both using the same AD account.
Would any of you SQL gurus out there know how to resolve this?
Thank youThe sp_delete_backuphistory job only deletes the historical info from the backup & restore tables in the msdb database. It does not do anything to the files on disk.
https://msdn.microsoft.com/en-us/library/ms188328.aspx
The code to actually purge the backup files from disk is inside the DatabaseBackup stored procedure. It is controlled by using the @CleanupTime parameter for that stored proc. Verify the value for that parameter is low enough to delete your files on disk. The
value is in hours, and I believe the default is 48 hours. The SQL agent job name usually starts with "DatabaseBackup...". -
Flat files data comma separated using SSIS.
Hi,
I have multiple flat files which come in comma separated columns. See example below :
Customer Data
CustID,FName,LName,Disease,Email,Phone
12345,Xyz,Smit,Bronchitis, Asthma and fever,[email protected],80000000
12346,Abc,Doe,fever Headache,[email protected],90000000
12347,Klu,joe,Sugar, cough and fever,[email protected],12345678
Please look at the ID's 12345 and 12347. The disease column has a internal comma space between. How do i remove the comma spaces in the disease column, so that it can be loaded from flat file to sql table using SSIS. ?
Please help !
ThanksHere is a full solution base on my post above (first option)
1. create temp table (Give it a unique name):
create table #T (Txt NVARCHAR(MAX))
GO
2. Insert all the data into temporary table. Each line in the text file, is a value for one column in a row in the table.
-- I will jump to the table and use simple insert.
-- If you have problem with step 1 then please inform us (this is simple bulk insert basically)
insert #T (Txt) values
('1234435,Xyz,Stemit,Brfsdonchitis, Asthma and fever,[email protected],80000000'),
('12346,Agjdfjbc,Doge,fevhhhher Headsxdshhache,[email protected],90000000'),
('123447,Klu,joe,Sugar, cough and fever,[email protected],12345678')
GO
the result should be like this:
Txt
1234435,Xyz,Stemit,Brfsdonchitis, Asthma and fever,[email protected],80000000
12346,Agjdfjbc,Doge,fevhhhher Headsxdshhache,[email protected],90000000
123447,Klu,joe,Sugar, cough and fever,[email protected],12345678
I use a SPLIT Function named Split_CLR_Fn. This is a CLR Split function that get input <string to split> and <string as delimiter,> and it return table with 2 columns ID, SplitData
For example if you use: SELECT * from Split_CLR_Fn('text1,text2,text3,',') then you get result:
ID SplitData
1 Text1
2 Text2
3 Text3
** You can find in the internet several good functions, I HIGHLY RECOMMENDED NOT TO USE T-SQL FUNCTIONS but CLR FUNCTION. Check thi link to understand why:
http://sqlperformance.com/2012/07/t-sql-queries/split-strings
** This is the best function that I know about and I use it, but I change the code a bit to return 2 columns and not just the SplitData as in this blog: http://sqlblog.com/blogs/adam_machanic/archive/2009/04/28/sqlclr-string-splitting-part-2-even-faster-even-more-scalable.aspx
That's it :-) we are ready for the solution which is very simple
Solution 1 (BAD solution but easy to write):
select
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 1) CustID,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 2) FName,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = 3) LName,
STUFF((select ',' + SplitData from Split_CLR_Fn(Txt,',') where ID > 3 and ID < (select MAX(ID) from Split_CLR_Fn(Txt,',')) - 1 for XML path('')), 1 , 1,'') Disease,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = (select MAX(ID) from Split_CLR_Fn(Txt,',')) - 1) Email,
(select SplitData from Split_CLR_Fn(Txt,',') where ID = (select MAX(ID) from Split_CLR_Fn(Txt,','))) Phone
from #T
GO
Solution 2: better in this case since the format is constant (this is the solution I wrote about above)
;With MyCTE as (
select
Txt,
SUBSTRING(Txt, 1, CHARINDEX(',', Txt, 1) - 1) as CustID
, SUBSTRING(
Txt
,CHARINDEX(',', Txt, 1) + 1 -- I start from the end of preview len
, CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)- CHARINDEX(',', Txt, 1) - 1
) as FName
, SUBSTRING(
Txt
,CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)+1 -- I start from the end of preview len
, CHARINDEX(',', Txt, CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1)+1) - CHARINDEX(',', Txt, CHARINDEX(',', Txt, 1)+1) - 1
) as LName
, RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1) as Phone
, RIGHT(LEFT(Txt, Len(Txt) - Len(RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1)) - 1), CHARINDEX(',', REVERSE(LEFT(Txt, Len(Txt) - Len(RIGHT(Txt, CHARINDEX(',', REVERSE(Txt), 1) - 1)) - 1)), 1) - 1) as Email
from #T
select CustID,FName,LName, Phone, Email, SUBSTRING(Txt, Len(CustID) + Len(FName) + Len(LName) + 4, Len(Txt) - Len(Email) - LEN(Phone) - Len(CustID) - Len(FName) - Len(LName) - 5) as Disease
from MyCTE
I hope that this is useful :-)
Ronen Ariely
[Personal Site] [Blog] [Facebook] -
I have just transferred all my raw, jpeg & tiff files into LR4 and using the opportunity to delete old raw files to free up disc space. How can i completely remove a raw file from my mac disc??
I'd be careful here: LR references files. It can copy them, like from an SD card to your hard drive, but it doesn't necessarily make duplicates. So when you say you "transferred" those files "into" LR it just means you referenced existing files. And if you delete those they're gone, unless you've made copies. LR does not import files by copying them into a managed library like Aperture does.
-
i hav iphone and i restart it and delete all the files inside how can i start it again it say i need a valid sim with no pin lock to activate iphone again
ICCID unknown is a critical failure, usually caused by hacking or jailbreaking the phone. It can not be fixed. The phone will need to be replaced.
-
Hello,
We are migrating the ssis packages from 2005 to 2012.
I'm unable to convert Activex Script Task from 2005 vesrsion to 2012 because in 2012 version Activex Script Task doesn't exist.
Can anyone please let me know what is the alternative way to convert Activex Script Task from 2005 vesrsion to 2012?Hi Vinay9738,
Have you tried to upgrade the SSIS 2005 packages to SSIS 2012 packages by using the SSIS Package Upgrade wizard? In certain cases, ActiveX script in SSIS won’t work and we need to either modify the script or replace the ActiveX Script with stock SSIS
tasks. You can find the mapping between some most common patterns used in DTS ActiveX Script and SSIS native tasks from the following link:
http://help.pragmaticworks.com/dtsxchange/scr/ActiveX%20Script%20Task.htm
Here is also a useful link about how to convert ADODB object of ActiveX Script to SSIS tasks:
http://help.pragmaticworks.com/dtsxchange/scr/FAQ%20-%20How%20to%20convert%20ADODB%20object%20of%20ActiveX%20Script%20to%20native%20SSIS%20Task.htm
Regards,
Mike Yin
TechNet Community Support -
Reading flat files with variable names in SSIS
I have a ssis package that reads a flat file from a network drive (using a flat file connection manager) and loads the data into sql server tables. I have this working on a fixed file name, however in reality the file name will not be the
same from run to run. Essentially, I need to check a folder each day and if there is a file there with a certain prefix (for example, 'datafile'), I need to process this file(s). In other words, if there is a file in the folder
called datafilexyz, my process needs to read it in and process it. If there are files named datafileabc, datafiledef, and testfile123, I need to read in and process the datafileabc and datafiledef flat files.
I'm not sure how to make this work. I haven't had any SSIS training, just what I can find on the internet and looking at existing SSIS packages (I haven't found any that do what I'm trying to do here) so I'm kind of lost.
Any help is appreciated.this is working well. How can I, after loading each flat file, move the flat file to an archive folder? I'm trying to use a file system task, but doing something wrong. I created and reference
an archive folder connection manager in the destination connection, and reference the original folder connection manager for the source connection, but get 'sourcepath is not valid on operation movefile'. I think a file system task is what
is needed (within the foreach loop after the data load for each flat file), but I'm not doing something correctly.
Sounds fine except for one thing. Did you assign the filename variable to connection string property of the source file connection manager used by file system task?
Also it should be existing file option you should choose for
source connection and existing folder for destination.
Please Mark This As Answer if it solved your issue
Please Mark This As Helpful if it helps to solve your issue
Visakh
My MSDN Page
My Personal Blog
My Facebook Page -
How to relink files inside an avchd package ?
I messed up a library and lost a project, then I found a copy of it in the backup folder, now when I open the project it asked me to relink files, but a part of them are inside an avchd package. How to relink those files ?
Thanksyou mean to normally import them or is there a specific reimport function ?
Thanks -
Trouble burning cd - delete old/temp files?
Hi there - I am having issues burning cds. Burning stops mid way due to an unknown error. I thought in the past, I rectified this by deleting old files, but cannot remember where to find them. Any suggestions on this? I am using the most recent version of iTunes, and have successfully run the cd burner diagnostics, with successful test results. Thanks!
Yes...in iTunes and I also tried burning via Sonic, and got the same result
-
My Macbook Pro won't auto delete old TM files
My wife's Macbook pro won't auto delete old files on the TM disc. I think it has to do with using a time capsule. I'd rather stop that and just use the WD Passport but how do I do that?
Hi..
My power cord has been trying to go out on me for a few weeks and tonight it just went capooey
If there's a power issue on your Mac, it's doubtful uninstalling and reinstalling software will make any difference.
Try Troubleshooting MagSafe adapters
If the adapter is ok, try Resetting the System Management Controller (SMC) -
Data convertion while exporting data into flat files using export wizard in ssis
Hi ,
while exporting data to flat file through export wizard the source table is having NVARCHAR types.
could you please help me on how to do the data convertion while using the export wizard?
Thanks.Hi Avs sai,
By default, the columns in the destination flat file will be non-Unicode columns, e.g. the data type of the columns will be DT_STR. If you want to keep the original DT_WSTR data type of the input column when outputting to the destination file, you can check
the Unicode option on the “Choose a Destination” page of the SQL Server Import and Export Wizard. Then, on the “Configure Flat File Destination” page, you can click the Edit Mappings…“ button to check the data types. Please see the screenshot:
Regards,
Mike Yin
TechNet Community Support -
Deleting Old Scratch Files Immediately
I've got QMaster working OK with Compressor so far. However, I had started a very large batch render on a Friday, only to come back to find the scratch disk full, with some files done and some not done. After investigating, I found out that the scratch disk (170GB free when I left) was full, and that none of the files were cleaned up after the successful jobs were finished, leaving a bunch of useless files on the drive, and causing the remaining files in the batch to die. I've got about 750GB of video to compress and send to a client, and I'd rather not have to do it one small batch at a time (24GB largest single file).
So, since this is not a matter of deleting files that are 1 day old (the lowest setting in the QMaster CP AFAIK), is there an easy way to clean out files that are no longer needed after a successful completion?
(Send a e-mail message on completion that will trigger a shell script? I can do that, but I'd really rather not have to, especially since the computers in the cluster are not my department's machines.)
p.s. I'd love to be able to avoid QMaster copying the originals to the scratch drive as well, say by sharing the volume, but that doesn't seem avoidable, and since it's automated, I'll live with itYou can try moving your cluster storage directory to a volume with more space. Where is your destination for the output files? The only time I see files in the cluster storage directory is when a batch failed. Running out of disk space would definitely cause a failure.
-
Delete old application files after update?
I downloaded the 3.4.1 update and want to delete the 3.4 application files to free up more space on my laptop's hard drive. Is this OK?
Also, what aboout deleting older versions of catalogs? Thanks.When you update, this removes the previous application files and replaces them with new ones. It will leave an empty folder.
If you no longer use a particular catalog you can delete it, but make sure you really don't want the catalog any more. Installing an update doesn't change the catalog only the application.
The only time a new catalog is created by LR (rather than by yourself), is when upgrading (rather than updating) from LR 2 to 3 and in this case LR creates a new catalog from your old one, leaving the old one intact. Once you are happy with everything you can remove the LR 2 catalog. Personally I don't. I kept the LR 1 catalog until LR 3 came out and will keep the LR 2 catalog until LR 4 comes out, but I am cautious like that and have loads of hard drives, so space isn't a problem. I just move the old catalog to a backup location...you never know what is going to happen and I may for some completely unknown reason need it some day.. -
How we will delete old recording files in ES2 ?.
Hi ,
The error ::
2010-02-26 16:25:16,811 ERROR [STDERR] Feb 26, 2010 4:25:16 PM com.adobe.idp.auditworkflow.dsc.service.storage.ProcessRecordingStorageImpl manageRecordingFileSizes
WARNING: deleting oldest recording(s) due to space limitation.
I am seeing above error many times in log file. Then googled i got http://forums.adobe.com/thread/506674?tstart=0 thread. But i know this one.
Is there any page in adminui , where i can delete all old recording at a shot.
Is there any way where i can improve buffer size of server ? Becuase some times i am getting error like below :
2010-02-26 16:12:12,558 WARN [com.adobe.idp.dsc.workmanager.workload.policy.MemoryPolicy] Memory threshold exceeded: maximum limit 95.0, current value 106.0
2010-02-26 16:12:12,559 WARN [com.adobe.idp.dsc.workmanager.workload.policy.WorkPolicyEvaluationBackgroundTask] Policies violated for statistics on 'adobejb_server1:wm_default'. Cause: com.adobe.idp.dsc.workmanager.workload.policy.WorkPolicyViolationException: Work policy violated. Will attempt to recover in 60000 ms
Please help me out !!Here's the help topic:
http://help.adobe.com/en_US/livecycle/9.0/workbenchHelp/000375.html#1548929
Here's how to configure the number of recordings to keep:
http://help.adobe.com/en_US/livecycle/9.0/workbenchHelp/001482.html#1608026
scott -
How do I get them back? It's from the Quicktime that already came with my Mac...please help me...It won't let me import movies to iMovie!!
What installer software package files? Could you tell me how to get there?
Afraid I'm not a mind reader. You have to download the installer packages from whatever source provides those installer packages. For instance, you would download the DivX installer package file from DivX. The Perian codec installer package would found at the Perian web site. QT "Pro" codec components installers may be include with older Motion or FCP Suite software or from the Apple site, etc., etc., etc. At this point all I know for sure is that that you have part of the DivX package installed and should reinstall the entire package if you plan to use it for anything.
Everyone's codec component configuration may be different depending on what codecs the need to use. Perhaps it would be easier if I provide an image showing the codec components installed on my Snow Leopard configured laptop:
This is a very basic set-up since I do most of my video work on my iMac which has FCP installed on it. In this case you can see:
a) AC3MovieImport.component (Apple)
b) AppleIntermediaCodec.component (Apple)
c) AppleMPEG2Codec.component (Apple)
d) Elgato Turbo.component (Elgato)
e) EyeTV MPEG Support.component (Elgato)
f) Flip4Mac WMV Advanced.component (Telestream)
g) Flip4Mac WMV Export.component (Telestream)
h) Flip4Mac WMV Import.component (Telestream)
i) Perian.component (Perian)
j) x264Encoder.component (MacUpdate)
The first three codecs were installed by my System Installer w/iMovie package (i.e., pre-installed on system when purchased or added afterwards by iMovie). The Elgato codec installers came with with my Turbo H.264 and EyeTV hardware devices. (I you don't have such devices, then you don't needcan't use these codecs.) The WMV codecs were purchased from Telestream. The Perian codec was a free download from the Perian web site and is used for DivX, XviD, 3ivX, DTS, AC3, MKV, FLV, and other audio/video/file type support. (I.e., I don't use the DivX package.) The X.264 codec is a leftover from my old Leopard system and not really used much anymore since I now prefer using HandBrake GUI for most X.264 custom encodes.
At this point, would assume you need to rerun the Snow Leopard/iMovie Installer if you purchased the system with system pre-installed or purchased the it as a system upgrade. Another alternative would be to find another Snow Leopard OS platform and simply copy the missing Apple iMovie codecs from a working system to your non-working system. All other codecs are up to you. Only you know what third-party codecs were installed on your system. And if you don't remember, you will probably find out what you are missing every time you try to play a video containing an unsupported codec.
Since you seem to be very inexperienced here, you may wish to consider joining a local user group or seeking some one-on-one assistance from someone in your local area.
Maybe you are looking for
-
ITunes on my iphone no longer connects to my airport express
My macbook air (OS10.8.5) connects to the airport express without any problems. My iphone 4S (OS8.1) has lost the connection.No idea with which update, i just noticed. All three are on the same WIFI network. I turned the airport and iphone off and on
-
Yosemite upgrade from Mavericks left me with a white screen.
i dont think the install completed And now my computer won't start up. It shows the apple icon and a progress bar and then just does nothing. I've turned it off and on a few time with the same result. Any ideas?
-
Ability to drag and drop to archive .mbox?
I used to be able to simply drag a mailbox from Mail into a location in the Finder, resulting in an .mbox file where I want it archived. I can no longer do this. I now have to use "Archive" within the app., I have to select where I want the archive p
-
How Can you Tell What BIOS you Have?
Is there anyway of finding out what BIOS version your drive has?
-
Bug Found: Illustrator CC *always* merges layers when opening a DXF file - CS6 works properly. When opening a DXF, CC merges multiple layers into one, even if you have the "Merge Layers" unchecked. Can a forum admin employee get this submitted as a