Recording live - large files

I recently recorded a band practice, using 8 channels. I ended up with an 8 GB Garageband file. I'm trying to go back and split up the file into the different individual songs. Each time I "save as" to save a new song, it takes 7-8 minutes. Is there a better way to do this? Or another program to record live and then import into GB?

everything GB does is "non-destructive", the files always remain intact unless you delete the entire region.
you can copy and paste between projects, and the pasted-to projects will be tiny because they contain no data just references to the original files in the original projects. however if you move files around or delete the original, the other files will break (more on that here:
http://www.bulletsandbones.com/GB/GBFAQ.html#saveasarchive )

Similar Messages

I need to update to the latest version of Snow Leapord-currently running v10.6. Because of where we live the slow download speed for such a large file has kept me from downloading the update. What can I do short of hooking up computer elsewhere?

I need to update to the latest version of Snow Leapord-currently running v10.6. Because of where we live the slow download speed for such a large file has kept me from downloading the update. What can I do short of hooking up computer elsewhere?

Do you ever visit a friend or realtive with a Mac who has faster internet? Maybe the local library has Macs on a fast line. If so, get a USB thumb drive and put this link on it from your computer:
http://support.apple.com/kb/DL1399
Then put this link on the drive:
http://support.apple.com/kb/DL1429
When you are at some place that has a Mac with a decent download speed. insert the thumb drive in that Mac and click on the first link ("DL1399") and direct the download to your thumb drive. Now do the same this with the second link.
The installer files will now be on the thumb drive and, when you get home, drag them from the thumb drive to your desktop. Install the Combo update first.

Q: HD players and carving up large files

Greetings,
First of all, this is what I'm using:
Intel(R) Core(TM)2 Duo CPU
8 gigs of RAM
External 2TB drive, internal 1TB drive (for media only)
Windows Vista, 64-bit
Adobe Premiere Pro CS4
First question, what is a good media player to play back MTS and M2TS files? I know there are several of them that come recommended on various sites, but I'm interested in knowing which ones to trust, which ones have the best options, and which ones some of you have had good luck with.
But my biggest problem is that a recent client gave me some very large files with which to work, and it's been like trying to shove an elephant through a drinking straw. They gave me an external hard drive with a ton of AVCHD footage on it (1440 x 1080, 29.97). No problem...I thought. They wanted to use the footage for a time lapse sequence of a big construction job of a unique dome they're building. The way they had described it to me, they were going to record the site over a long period of time, programming the camera to capture a few seconds every few hours.
Well, someone screwed the pooch. The camera was turned on almost every day, but they let the thing run all day! So, some of these files are over 30 gigs, and it's been murder trying to import them and get them to play. As I expected, it was pointless to try to use the external hard drive as a source since it's not even firewire but USB, and getting the files to play back in PP was not a good experience; I could practically hear the drive choking to death. But even when I transferred a couple of large files to my internal terabyte, they still played back terribly in PP, in both the viewer and on the timeline. When I moved the sequence marker, it would hang up the system for long periods of time, and then playback was choppy at best.
I imagine it could be processor speed - or lack thereof. I know a quad would be better, but I've played back smaller MTS files with no problem, and right now getting more processor speed isn't an option - if that's even the main problem. The thing that adds insult to injury is that this project is going to mix SD (DVCAM, I think) with this AVCHD, so I don't even think shooting the timelapse stuff in HD was even necessary (they had a small Sony camera they wanted to use). I know I could convert the m2ts files to AVIs or something like that, but that would take forever since I'm going to need to use a little chunk of each file (to edit the time lapse).
So, my second question is, is it possible to carve up these large files somehow, so that I can import smaller files into PP and use the (source) viewer in PP to scan the footage more easily? Like I said, it's been difficult to scan a 7-hour, 35 gig clip without missing something and/or hanging up the system. I don't have that problem with smaller source files. Unfortunately, I have a feeling carving up these m2ts isn't possible without converting the files to another format....?
I don't know, maybe it isn't the files and it has something else to do with my computer. But I've been editing SD without much trouble at all, and also some HD (with smaller files), so something tells me these files are just too big. My drive is 7200 rpm, and I don't have anything else on this computer except some other Adobe stuff (PowerPoint, Reader; the "garbage" is minimal. Anybody have any ideas? Thanks in advance.
Paul

I would be declining to take on this project on the basis of what you have told us about what you
have been supplied with...unless they are prepared to acknowledge the scale of the task (problem) and pay on an hourly basis.
Its going to take a lot of time!

Network speed affected by large file copy operations. Also, why intermittent network outages?

Hi
I have a couple of issues on our company network.
The first is thate a single large file copy imapcts the entire network and dramatically reduces network speed and the second is that there are periodic outages where file open/close/save operations may appear to hang, and also where programs that rely on
network connectivity e.g. email, appear to hang. It is as though the PC loses it's connection to the network, but the status of the network icon does not change. For the second issue if we wait the program will respond but the wait period can be up to 1min.
The downside of this is that this affects Access databases on our server so that when an 'outage' occurs the Access client cannot recover and hangs permamnently.
We have a Windows Active Directory domain that comprises Windows 2003 R2 (soon to be decommissioned), Windows Server 2008 Standard and Windows Server 2012 R2 Standard domain controllers. There are two member servers: A file server running Windows 2008 Storage
Server and a remote access server (which also runs WSUS) running Windows Server 2012 Standard. The clients comprise about 35 Win7 PC's and 1 Vista PC.
When I copy or move a large file from the 2008 Storage Server to my Win7 client other staff experience massive slowdowns when accessing the network. Recently I was moving several files from the Storage Server to my local drive. The files comprised pairs
(e.g. folo76t5.pmm and folo76t5.pmi), one of which is less than 1MB and the other varies between 1.5 - 1.9GB. I was moving two files at a time so the total file size for each operation was just under 2GB.
While the file move operation was taking place a colleague was trying to open a 36k Excel file. After waiting 3mins he asked me for help. I did some tests and noticed that when I was not copying large files he could open the Excel file immediately. When
I started copying more data from the Storage Server to my local drive it took several minutes before his PC could open the Excel file.
I also noticed on my Win7 client that our email client (Pegasus Mail), which was the only application I had open at the time would hang when the move operation was started and it would take at least a minute for it to start responding.
Ordinarlily we work with many files
Anyone have any suggestions, please? This is something that is affecting all clients. I can't carry out file maintenance on large files during normal work hours if network speed is going to be so badly impacted.
I'm still working on the intermittent network outages (the second issue), but if anyone has any suggestions about what may be causing this I would be grateful if you could share them.
Thanks

What have you checked for resource usage during one of these copies of a large file?
At a minimum I would check Task Manager>Resource Monitor. In particular check the disk and network usage. Also, look at RAM and CPU while the copy is taking place.
What RAID level is there on the file server?
There are many possible areas that could be causing your problem(s). And it could be more than one thing. Start by checking these things. And go from there.
Hi, JohnB352
Thanks for the suggestions. I have monitored the server and can see that the memory is nearly maxed out with a lot of hard faults (varies between several hundred to several thousand), recorded during normal usage. The Disk and CPU seem normal.
I'm going to replace the RAM and double it up to 12GB.
Thanks! This may help with some other issues we are having. I'll post back after it has been done.
[Edit]
Forgot to mention: there are 6 drives in the server. 2 for the OS (Mirrored RAID 1) and 4 for the data (Striped RAID 5).

Windows Explorer misreads large-file .zip archives

   I just spent about 90 minutes trying to report this problem through
the normal support channels with no useful result, so, in desperation,
I'm trying here, in the hope that someone can direct this report to some
useful place.
   There appears to be a bug in the .zip archive reader used by Windows
Explorer in Windows 7 (and up, most likely).
   An Info-ZIP Zip user recently reported a problem with an archive
created using our Zip program. The archive was valid, but it contained
a file which was larger than 4GiB. The complaint was that Windows
Explorer displayed (and, apparently believed) an absurdly large size
value for this large-file archive member. We have since reproduced the
problem.
   The original .zip archive format includes uncompressed and compressed
sizes for archive members (files), and these sizes were stored in 32-bit
fields. This caused problems for files which are larger than 4GiB (or,
on some system types, where signed size values were used, 2GiB). The
solution to this fundamental limitation was to extend the .zip archive
format to allow storage of 64-bit member sizes, when necessary. (PKWARE
identifies this format extension as "Zip64".)
   The .zip archive format includes a mechanism, the "Extra Field", for
storing various kinds of metadata which had no place in the normal
archive file headers. Examples include OS-specific file-attribute data,
such as Finder info and extended attributes for Apple Macintosh; record
format, record size, and record type data for VMS/OpenVMS; universal
file times and/or UID/GID for UNIX(-like) systems; and so on. The Extra
Field is where the 64-bit member sizes are stored, when the fixed 32-bit
size fields are too small.
   An Extra Field has a structure which allows multiple types of extra
data to be included. It comprises one or more "Extra Blocks", each of
which has the following structure:
       Size (bytes) | Description
      --------------+------------
            2       | Type code
            2       | Number of data bytes to follow
        (variable) | Extra block data
   The problem with the .zip archive reader used by Windows Explorer is
that it appears to expect the Extra Block which includes the 64-bit
member sizes (type code = 0x0001) to be the first (or only) Extra Block
in the Extra Field. If some other Extra Block appears at the start of
the Extra Field, then its (non-size) data are being incorrectly
interpreted as the 64-bit sizes, while the actual 64-bit size data,
further along in the Extra Field, are ignored.
   Perhaps the .zip archive _writer_ used by Windows Explorer always
places the Extra Block with the 64-bit sizes in this special location,
but the .zip specification does not demand any particular order or
placement of Extra Blocks in the Extra Field, and other programs
(Info-ZIP Zip, for example) should not be expected to abide by this
artificial restriction. For details, see section "4.5 Extensible data
fields" in the PKWARE APPNOTE:
      http://www.pkware.com/documents/casestudies/APPNOTE.TXT
   A .zip archive reader is expected to consider the Extra Block type
codes, and interpret accordingly the data which follow. In particular,
it's not sufficient to trust that any particular Extra Block will be the
first one in the Extra Field. It's generally safe to ignore any Extra
Block whose type code is not recognized, but it's crucial to scan the
Extra Field, identify each Extra Block, and handle it according to its
type.
   Here are some relatively small (about 14MiB each) test archives which
illustrate the problem:
      http://antinode.info/ftp/info-zip/ms_zip64/test_4g.zip
      http://antinode.info/ftp/info-zip/ms_zip64/test_4g_V.zip
      http://antinode.info/ftp/info-zip/ms_zip64/test_4g_W.zip
   Correct info, from UnZip 6.00 ("unzip -lv"):
Archive: test_4g.zip
Length   Method    Size Cmpr    Date    Time   CRC-32   Name
4362076160 Defl:X 14800839 100% 05-01-2014 15:33 6d8d2ece test_4g.txt
Archive: test_4g_V.zip
Length   Method    Size Cmpr    Date    Time   CRC-32   Name
4362076160 Defl:X 14800839 100% 05-01-2014 15:33 6d8d2ece test_4g.txt
Archive: test_4g_W.zip
Length   Method    Size Cmpr    Date    Time   CRC-32   Name
4362076160 Defl:X 14800839 100% 05-01-2014 15:33 6d8d2ece test_4g.txt
(In these reports, "Length" is the uncompressed size; "Size" is the
compressed size.)
   Incorrect info, from (Windows 7) Windows Explorer:
Archive        Name          Compressed size   Size
test_4g.zip    test_4g.txt         14,454 KB   562,951,376,907,238 KB
test_4g_V.zip test_4g.txt         14,454 KB   8,796,110,221,518 KB
test_4g_W.zip test_4g.txt         14,454 KB   1,464,940,363,777 KB
   Faced with these unrealistic sizes, Windows Explorer refuses to
extract the member file, for lack of (petabytes of) free disk space.
   The archive test_4g.zip has the following Extra Blocks: universal
time (type = 0x5455) and 64-bit sizes (type = 0x0001). test_4g_V.zip
has: PWWARE VMS (type = 0x000c) and 64-bit sizes (type = 0x0001).
test_4g_W.zip has: NT security descriptor (type = 0x4453), universal
time (type = 0x5455), and 64-bit sizes (type = 0x0001). Obviously,
Info-ZIP UnZip has no trouble correctly finding the 64-bit size info in
these archives, but Windows Explorer is clearly confused. (Note that
"1,464,940,363,777 KB" translates to 0x0005545500000400 (bytes), and
"0x00055455" looks exactly like the size, "0x0005" and the type code
"0x5455" for a "UT" universal time Extra Block, which was present in
that archive. This is consistent with the hypothesis that the wrong
data in the Extra Field are being interpreted as the 64-bit size data.)
   Without being able to see the source code involved here, it's hard to
know exactly what it's doing wrong, but it does appear that the .zip
reader used by Windows Explorer is using a very (too) simple-minded
method to extract 64-bit size data from the Extra Field, causing it to
get bad data from a properly formed archive.
   I suspect that the engineer involved will have little trouble finding
and fixing the code which parses an Extra Field to extract the 64-bit
sizes correctly, but if anyone has any questions, we'd be happy to help.
   For the Info-ZIP (http://info-zip.org/) team,
   Steven Schweda

> We can't get the source (info-zip) program for test.
   I don't know why you would need to, but yes, you can:
      http://www.info-zip.org/
      ftp://ftp.info-zip.org/pub/infozip/src/
You can also get pre-built executables for Windows:
      ftp://ftp.info-zip.org/pub/infozip/win32/unz600xn.exe
      ftp://ftp.info-zip.org/pub/infozip/win32/zip300xn.zip
> In addition, since other zip application runs correctly. Since it should
> be your software itself issue.
   You seem to misunderstand the situation. The facts are these:
   1. For your convenience, I've provided three test archives, each of
which includes a file larger than 4GiB. These archives are valid.
   2. Info-ZIP UnZip (version 6.00 or newer) can process these archives
correctly. This is consistent with the fact that these archives are
valid.
   3. Programs from other vendors can process these archives correctly.
I've supplied a screenshot showing one of them (7-Zip) doing so, as you
requested. This is consistent with the fact that these archives are
valid.
   4. Windows Explorer (on Windows 7) cannot process these archives
correctly, apparently because it misreads the (Zip64) file size data.
I've supplied a screenshot of Windows Explorer showing the bad file size
it gets, and the failure that occurs when one tries to use it to extract
the file from one of these archives, as you requested. This is
consistent with the fact that there's a bug in the .zip reader used by
Windows Explorer.
   Yes, "other zip application runs correctly." Info-ZIP UnZip runs
correctly. Only Windows Explorer does _not_ run correctly.

Problems copying large files to external disks

I have a lot of media and so multiple USB and network based external hard disks.
I'm having trouble with two in particular that are recent buys. I initially mounted them via a USB hub onto my Time Capsule, but when I had errors, I've now tried mounting them directly to my MacBook Air - and for comparison directly to a Windows laptop.
And I'm only having problems when it's a Mac doing the copying (MBA to either USB mounted via Time Capsule or directly USB mounted on the MBA).
The problem is that the drive appears to behave OK for initial copies - but I'm trying to put a set of old movies (captured from a VCR ages ago that I'd recorded off TV) onto one of the drives and (a) it takes ages to copy and (b) eventually I get a write failure. The specific error message is
The Finder can't complete the operation because some data in "" can't be written. (Error code -36)
I"ve tried a whole variety of setups - as I've said via Time Capsule and directly mounted. I also wondered if the file system on the drive would make a difference. Out of the box it was formatted with FAT32 and when that failed I've now reformatted with MacOS file system - which I would have thought would give better compatibility (and I've read that FAT32 has a large file size limit - although I think it's 4GB and while I do have one file at 4.04GB, it's not failing the copy on that file).
I've also connected the drive (when formatted FAT32) to a Windows laptop and (a) it copies faster and (b) it copies successfully.
Any suggestions for this? Is there some kind of large file (all are >1GB) copy issue with USB mounted devices in OSX?

As I mentioned in my original post while the disks were originally formatted FAT32 I reformatted then and changed them to Mac OS Extended (Journaled) so that isn't the issue. I still have the problem with the disks using Apple format.
I've noticed that if I do the copy in pieces, it seems to work. I.e. if I select the full set of 45GB in one copy and paste operation (dragging and dropping using Finder) it fails part way through.
But if I copy 3/4 movies at a time going back until I have copied all of them then I can get all of them on the disk.
It suggests to me it's some kind of issue with copying and pasting a very large amount of data in Snow Leopard?

File Splitting for Large File processing in XI using EOIO QoS.

Hi
I am currently working on a scenario to split a large file (700MB) using sender file adapter "Recordset Structure" property (eg; Row, 5000). As the files are split and mapped, they are, appended to a destination file. In an example scenario a file of 700MB comes in (say with 20000 records) the destination file should have 20000 records.
To ensure no records are missed during the process through XI, EOIO, QoS is used. A trigger record is appended to the incoming file (trigger record structure is the same as the main payload recordset) using UNIX shellscript before it is read by the Sender file adapter.
XPATH conditions are evaluated in the receiver determination to eighther append the record to the main destination file or create a trigger file with only the trigger record in it.
Problem that we are faced is that the "Recordset Structure" (eg; Row, 5000) splits in the chunks of 5000 and when the remaining records of the main payload are less than 5000 (say 1300) those remaining 1300 lines get grouped up with the trigger record and written to the trigger file instead of the actual destination file.
For the sake of this forum I have a listed a sample scenario xml file representing the inbound file with the last record wih duns = "9999" as the trigger record that will be used to mark the end of the file after splitting and appending.
<?xml version="1.0" encoding="utf-8"?>
<ns:File xmlns:ns="somenamespace">
<Data>
     <Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
     <Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Cage_Code>"3NQN1"</Cage_Code>
          <Extract_Code>"A"</Extract_Code>
     </Row>
</Data>
</ns:File>
In the sender file adapter I have for test purpose changed the "Recordset structure" set as "Row,5" for this sample xml inbound file above.
I have two XPATH expressions in the receiver determination to take the last record set with the Duns = "9999" and send it to the receiver (coominication channel) to create the trigger file.
In my test case the first 5 records get appended to the correct destination file. But the last two records (6th and 7th record get sent to the receiver channel that is only supposed to take the trigger record (last record with Duns = "9999").
Destination file: (This is were all the records with "Duns NE "9999") are supposed to get appended)
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
     <R3Row>
          <Duns>"001001924"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001925"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
     <R3Row>
          <Duns>"001001926"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</xtract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001927"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"001001928"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Extract_Code>"A"</Extract_Code>
     </R3Row>
</R3File>
Trigger File:
<?xml version="1.0" encoding="UTF-8"?>
<R3File>
          <R3Row>
          <Duns>"001001929"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
          <R3Row>
          <Duns>"9999"</Duns>
          <Duns_Plus_4>""</Duns_Plus_4>
          <Ccr_Extract_Code>"A"</Ccr_Extract_Code>
     </R3Row>
</R3File>
I ve tested the XPATH condition in XML Spy and that works fine. My doubts are on the property "Recordset structure" set as "Row,5".
Any suggestions on this will be very helpful.
Thanks,
Mujtaba

Hi Debnilay,
We do have 64 bit architecture and still we have the file processing problem. Currently we are splitting the file into smaller chuncks and processsing. But we want to process as a whole file.
Thanks
Steve

Determine the number of records in a file

I want to analyse record types from a large input file and so far have managed to extract different groups and created the relevant files. All I want to do now is count the number of records in each file so I can produce a report showing the different types but cannot figure out how to do this, I know it will be simple but I cannot get any command to accomplish this, help please!

in applescript, there are a few ways to count patterned text. if (as jeffrey suggests) each record is a separate line, then you can use the following:
set textContents to read "/path/to/file.txt"
count of paragraphs of textContents
if records are determined by some other process (such as a special delimiting character or phrase) you can use text items (replace the @ sign with the special character or phrase):
set textContents to read "/path/to/file.txt"
set {oldTID, my text item delimiters} to {my text item delimiters, "@"}
set theCount to count of text items of textContents
set my text item delimiters to oldTID

Flash media server taking forever to load large files

We purchased FMIS and we are encoding large 15+ hour MP4 recordings using flash media encoder. When opening these large files for playback, which have not been opened recently the player displays the loading indicator for up to 4 minutes! Once it has apparently been cached on the server it opens immediately from any browser even after clearing local browser cache. So a few questions for the experts
1. Why is it taking so long to load the file. Is it because the MP4 metadata is in the wrong format and the file is so huge? I read somewhere that Media Encoder records with incorrect MP4 metadata is that still the case?
2. Once its cached on the server, exactly how much of it is cached. Some of these files are larger than 500mb.
3. What fms settings do you suggest I change. FMIS is running on windows server R2 64 bit, but FMIS itself is 32 bit. We have not upgraded to the 64 bit version. We have 8GB of ram. Is it OK to set FMS cache to 3GB. And would that only have enough room for 3-4 large files, because we have hundreds of them.
best,
Tuviah
Lead programmer, solid state logic inc

Hi Tuviah,
You may want to email me offline about more questions here as it can get a little specific but I'll hit the general problems here.
MP4 is a fine format, and I won't speak ill of it, but it does have weaknesses. In FMS implementation those weaknesses tend to manifest around the combination of recording and very large files, so some of these things are a known issue.
The problem is that MP4 recording is achieved through what's called MP4 fragmentation. It's a part of the MP4 spec that not every vendor supports, but has a very particular purpose, namely the ability to continually grow an MP4 style file efficiently. Without fragments one has the problem that a large file must be constantly rewritten as a whole for updating the MOOV box (index of files) - fragments allow simple appending. In other words it's tricky to make mp4 recording scalable (like for a server ) and still have the basic MP4 format - so fragments.
There's a tradeoff to this however, in that the index of the file is broken up over the whole file. Also likely these large files are tucked away on a NAS for you or something similar. Normal as you likely can't store all of them locally. However that has the bad combo of needing to index the file (touching parts of the whole thing) and doing network reads to do it. This is likely the cause of the long delay you're facing - here are some things you can do to help.
1. Post process the F4V/MP4 files into non fragmented format - this may help significantly in load time, though it could still be considered slow it should increase in speed. Cheap to try it out on a few files. (F4V and MP4 are the same thing for this purpose - so don't worry about the tool naming)
http://www.adobe.com/products/flashmediaserver/tool_downloads/
2. Alternatively this is why we created the raw: format. For long recording mp4 is just unideal and raw format solves many of the problems involved in doing this kind of recording. Check it out
http://help.adobe.com/en_US/flashmediaserver/devguide/WSecdb3a64785bec8751534fae12a16ad027 7-8000.html
3. You may also want to check out FMS HTTP Dynamic Streaming - it also solves this problem, along with others like content protection and DVR and it's our most recent offering in tech, so it has a lot of strengths the other areas don't.
http://www.adobe.com/products/httpdynamicstreaming/
Hope that helps,
Asa

Handling Large files in PI scenarios?

Hello,
We have lot of scenarios (almost 50) where we deal with file interfaces atleast in receiver or sender side. Some of them are just file transfers where we use AAE and some are where we have to do message mapping (sometimes very complex ones).
the interfaces work perfectly fine will a normal file which dont have much records but recently we started testing big files with over 1000 records and its taking a lot of time to process. It is also causing other messages which gets lined up in the same queue to wait in the queue for the amount of time it takes for the first message to process.
This must be a very practical scenario where PI has to process large files specially files coming from banks. What is the best way to handle its processing? Apart from having a better system hardware (we are currently in the test environment. Production environment will definetely be better) is there any technique which might help us improve the processing of large files without data loss and without interrupting other message?
Thanks,
Yash

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

(urgent) SQL*Loader Large file support in O734

hi there,
i have the following sqlloader error when trying to upload data file(s),
each has size 10G - 20G to Oracle 734 DB on SunOS 5.6 .
>>
SQL*Loader-500: Unable to open file (..... /tstt.dat)
SVR4 Error: 79: Value too large for defined data type
<<
i know there's bug fix for large file support in Oracle 8 -
>>
Oracle supports files over 2GB for the oracle executable.
Contact Worldwide Support for information about fixes for bug 508304,
which will add large file support for imp, exp, and sqlldr
<<
however, really want to know if any fix for Oracle 734 ?
thx.

Example
Control file
C:\DOCUME~1\MAMOHI~1>type dept.ctl
load data
infile dept.dat
into table dept
append
fields terminated by ',' optionally enclosed by '"'
trailing nullcols
(deptno integer external,
dname char,
loc char)
Data file
C:\DOCUME~1\MAMOHI~1>type dept.dat
50,IT,VIKARABAD
60,INVENTORY,NIZAMABAD
C:\DOCUME~1\MAMOHI~1>
C:\DOCUME~1\MAMOHI~1>dir dept.*
Volume in drive C has no label.
Volume Serial Number is 9CCC-A1AF
Directory of C:\DOCUME~1\MAMOHI~1
09/21/2006 08:33 AM               177 dept.ctl
04/05/2007 12:17 PM                41 dept.dat
               2 File(s)          8,043 bytes
               0 Dir(s)   1,165 bytes free
Intelligent sqlldr command
C:\DOCUME~1\MAMOHI~1>sqlldr userid=hary/hary control=dept.ctl
SQL*Loader: Release 10.2.0.1.0 - Production on Thu Apr 5 12:18:26 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Commit point reached - logical record count 2
C:\DOCUME~1\MAMOHI~1>sqlplus hary/hary
SQL*Plus: Release 10.2.0.1.0 - Production on Thu Apr 5 12:18:37 2007
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
As I am appending I got two extra rows. One department in your district and another in my district :)
SQL> select * from dept;
    DEPTNO DNAME          LOC
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 OPERATIONS     BOSTON
        50 IT             VIKARABAD
        60 INVENTORY      NIZAMABAD
6 rows selected.
SQL>

Process large file using BPEL

My project have a requirement of processing large file (10 MB) all at once. In the project, the file adapter reads the file, then calls 5 other BPEL process to do 10 different validations before delivering to oracle database. I can't use debatch feature of adapter because of Header and detail record validation requirement. I did some performace tuing (eg: auditlevel to minimum, logging level to error, JVM size to 2GB etc..) as per performance tuing specified in Oracle BPEL user guide. We are using 4 CPU, 4GB RAM IBM AIX 5L server. I observed that the Receive activity in the begining of each process is taking lot of time, while other transient process are as per expected.
Following are statistics for receive activity per BPEL process:
500KB: 40 Sec
3MB: 1 Hour
Because we have 5 BPEL process, so lot of time is wasted in receive activity.
I did't try 10 MB so far, because of poor performance figure for 3 MB file.
Does any one have any idea how to improve performance of begining receive activity of BPEL process?
Thanks
-Simanchal

I believe the limit in SOA Suite is 7MB if you want to use the full payload and perform some kind of orchastration. Otherwise you need to do some kind of debatching, which you stated will not work.
SOA Suite is not really designed for your kind of use case as it needs to parocess this file in memory, when any transformation occurs it can increase this message between 3 - 10 times. If you are writing to a database why can you read the rows one by one?
If you are wanting to perform this kind of action have a look at ODI (Oracle Data Integrator). I Also believe that OSB (Aqua Logic) can handle files upto 200MB this this can be an option as well, but it may require debatching.
cheers
James

Large file doesn't work in a message-splitting scenario

Hello,
I'm trying to measure the XI performance in a message-splitting scenario by having the file adapter to pull the xml file below and send it to XI to perform a simple "message split w/o BPM" and generate xml files for each record in another folder.   I tried 100, 500, 1000 records and they all turned out fine. Once I tried the 2000-records case, the status of that message is WAITING in RWB and I checked sxmb_moni, the message is in "recorded for outbound messaging", I couldn't find any error.
Is there some kinda of threshold that can be adjusted for large files? Thank you.
<?xml version="1.0" encoding="UTF-8"?>
<ns0:MultiXML xmlns:ns0="urn:himax:b2bi:poc:multiMapTest">
   <Record>
      <ID>1</ID>
      <FirstName>1</FirstName>
      <LastName>1</LastName>
   </Record>
</ns0:MultiXML>

Hello,
The Queue ID is "XBTO1__0000". I double-clicked it, it took me to the "qRFC Monitor (Inbound Queue) screen, and showed a XBT01_0000 queue with 3 Entries.
I double-clicked on that, and it took me to another screen that showed a queue with same same and "Running" in Status.
I double-clicked on that, and i saw three entries. For the latest entry, its StatusText says "Transaction recorded".
I double-clicked on that, it took me to Function Builder:Display SXMS_ASYNC_EXEC.
What do I from here?

UTL_FILE.get_line won't read large files ?

I am trying to read a large fixed length flat file. If I cut the file down to really small it will read it but it reads it as a single line. If I try to read a larger file > 32k I get a READ_ERROR. I am pretty sure it has to do with the end of line marker but I saw nothing about that in the UTL_FILE documentation. This is on UNIX, new line character after each record in the file. Standard unix flat file.
Any ideas on what to do?
Thanks in advance
Matt
[email protected]
my code:
BEGIN
BEGIN
std_file := UTL_FILE.FOPEN('&4','&1','r',32767);
EXCEPTION
WHEN UTL_FILE.INVALID_PATH THEN
RAISE_APPLICATION_ERROR(-20011,'Invalid Path for STD file, &4/&1');
WHEN OTHERS THEN
RAISE_APPLICATION_ERROR(-20014,'Other Error trying to open STD file, &4/&1');
END;
IF UTL_FILE.is_open(std_file) = FALSE THEN
RAISE_APPLICATION_ERROR(-20015,'Could not open STD file, &4/&1');
END IF;
-- READ STD FILE HEADER
BEGIN
UTL_FILE.get_line(std_file,hdr_text);
EXCEPTION
WHEN UTL_FILE.INVALID_FILEHANDLE THEN
RAISE_APPLICATION_ERROR(-20017,'STD read file handle not valid');
WHEN UTL_FILE.INVALID_OPERATION THEN
RAISE_APPLICATION_ERROR(-20018,'STD read invalid operation error');
WHEN UTL_FILE.READ_ERROR THEN
RAISE_APPLICATION_ERROR(-20019,'STD read error');
WHEN NO_DATA_FOUND THEN
RAISE_APPLICATION_ERROR(-20020,'STD read no data found');
WHEN VALUE_ERROR THEN
RAISE_APPLICATION_ERROR(-20021,'STD read value error');
END;
-- PROCESS TRANSACTIONS
LOOP
BEGIN
tran_text := NULL;
UTL_FILE.get_line(std_file,tran_text);
EXCEPTION
WHEN no_data_found THEN EXIT; -- EOF
WHEN value_error THEN
RAISE_APPLICATION_ERROR(-20010,'STD record too long.');
END;
std_rowcount := std_rowcount + 1;
END LOOP;
UTL_FILE.FCLOSE(std_file);
EXCEPTION
WHEN NO_DATA_FOUND THEN
RAISE_APPLICATION_ERROR(-20001,'No Data Found.');
WHEN UTL_FILE.INVALID_PATH THEN
RAISE_APPLICATION_ERROR(-20002,'Invalid Path ');
WHEN UTL_FILE.INVALID_MODE THEN
RAISE_APPLICATION_ERROR(-20003,'Invalid Mode ');
WHEN UTL_FILE.INVALID_OPERATION THEN
RAISE_APPLICATION_ERROR(-20004,'Invalid Operation ');
END;
null

We are still hung up on this. I tried implementing the code from STEVE'S XML book but still haven't resovled it.
The clob is being created via XSU see below. The new char[8192] appeasr to force the output file to 8K
with trailing characters on small clobs but adds a carraige return each 8K on larger ones.
As usual any input is appreciated from all. Doese anyone know of a good JAVA forum like this one?
Thanks
PROCEDURE BuildXml(v_return OUT INTEGER, v_message OUT VARCHAR2,string_in VARCHAR2,xml_CLOB OUT NOCOPY CLOB) IS
queryCtx DBMS_XMLquery.ctxType;
Buffer RAW(1024);
Amount BINARY_INTEGER := 1024;
Position INTEGER := 1;
sql_string VARCHAR2(2000) := string_in;
BEGIN
v_return := 1;
v_message := 'BuildXml completed succesfully.';
queryCtx := DBMS_XMLQuery.newContext(sql_string);
xml_CLOB := DBMS_XMLQuery.getXML(queryCtx);
DBMS_XMLQuery.closeContext(queryCtx);
EXCEPTION WHEN OTHERS THEN
v_return := 0;
v_message := 'BuildXml failed - '||SQLERRM;
END BuildXml;
create or replace and compile java source named sjs.write_CLOB as
import java.io.*;
import java.sql.*;
import java.math.*;
import oracle.sql.*;
import oracle.jdbc.driver.*;
public class write_CLOB extends Object
public static void pass_str_array(oracle.sql.CLOB p_in,java.lang.String f_in)
throws java.sql.SQLException, IOException
File target = new File(f_in);
FileWriter fw = new FileWriter(target);
BufferedWriter out = new BufferedWriter(fw);
Reader is = p_in.getCharacterStream();
char buffer[] = new char[8192];
int length;
while( (length=is.read(buffer)) != -1) {
out.write(buffer);
is.close();
fw.close();
/

How to send a large file in XI ?

How to send a large file in XI ?

hi,
use a splitting mechanism to convert large files into smaller files and process them as if they where independent.
You can use the "Recordsets Per Message" parameter in the File Adapter sender comm channel by using File Content Conversion, to create a new message for each 1000,10000,etc records in the source file...this could be a way of splitting.
If you scenario works ok for small files, maybe you can develop another scenario that runs previous the current one, that only splits files (e.g. File-XI-File) and then puts the output smaller files in the directory that your current scenario monitors.
Check this blog for huge file processing
/people/alessandro.guarneri/blog/2007/02/21/sap-xi-acting-as-a-huge-file-mover
Thanks,
Vijaya

Recording live - large files

Similar Messages

Maybe you are looking for