Store large volume of Image files, what is better ? File System or Oracle

I am working on a IM (Image Management) software that need to store and manage over 8.000.000 images.
I am not sure if I have to use File System to store images or database (blob or clob).
Until now I only used File System.
Could someone that already have any experience with store large volume of images tell me what is the advantages and disadvantages to use File System or to use Oracle Database ?
My initial database will have 8.000.000 images and it will grow 3.000.000 at year.
Each image will have sizes between 200 KB and 8 MB, but the mean is 300 KB.
I am using Oracle 10g I. I read in others forums about postgresql and firebird, that isn't good store images on database because always database crashes.
I need to know if with Oracle is the same and why. Can I trust in Oracle for this large service ? There are tips to store files on database ?
Thank's for help.
Best Regards,
Eduardo
Brazil.

1) Assuming I'm doing my math correctly, you're talking about an initial load of 2.4 TB of images with roughly 0.9 TB added per year, right? That sort of data volume certainly isn't going to cause Oracle to crash, but it does put you into the realm of a rather large database, so you have to be rather careful with the architecture.
2) CLOBs store Character Large OBjects, so you would not use a CLOB to store binary data. You can use a BLOB. And that may be fine if you just want the database to be a bit-bucket for images. Given the volume of images you are going to have, though, I'm going to wager that you'll want the database to be a bit more sophisticated about how the images are handled, so you probably want to use [Oracle interMedia|http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14302/ch_intr.htm#IMURG1000] and store the data in OrdImage columns which provides a number of interfaces to better manage the data.
3) Storing the data in a database would generally strike me as preferrable if only because of the recoverability implications. If you store data on a file system, you are inevitably going to have cases where an application writes a file and the transaction to insert the row into the database fails or a the transaction to delete a row from the database succeeds before the file is deleted, which can make things inconsistent (images with nothing in the database and database rows with no corresponding images). If something fails, you also can't restore the file system and the database to the same point in time.
4) Given the volume of data you're dealing with, you may want to look closely at moving to 11g. There are substantial benefits to storing large objects in 11g with Advanced Compression (allowing you to compress the data in LOBs automatically and to automatically de-dupe data if you have similar images). SecureFile LOBs can also be used to substantially reduce the amount of REDO that gets generated when inserting data into a LOB column.
Justin

Similar Messages

Managing large volumes of images

I have a large volume of images. Mostly in raw format. I almost lost them all a few years ago when something happened to iPhoto 06. Since that time I avoided iPhoto and have been managing the file structure myself and using Lightroom.
All my images are now stored on a NAS running Raid 0. I am feeling a little more secure now, so....
...I am interested to know what database improvements have been made to iPhoto. Is it save to use with that much data? Does it work well with Lightroom? How does it work with Aperture or does Aperture just replace iPhoto? Can the iPhoto or Aperture database reside on my NAS?
Cheers.

1. The protection against any database failure is a good current back up. iPhoto makes a automatic back up of the database file. This facilitates recovery from issues. However this is not a substitute for a back up of the entire Library.
2. The number of images is what's important to iPhoto, not the total file size. iPhoto is good for 250k images and we've seen reports on here from folks with more than 100,000. So it will work with volume.
3. It doesn't work with Lightroom at all. This is germane to Aperture as well.
iPhoto, Lightroom and Aperture are all essentially Database applications. All of them want to manage the data and will only share the data under certain protocols.
Aperture and Lightroom don't actually edit photos. When you process a photo in these apps, the file is not changed. Instead, the decisions are recorded in the database and applied live when you view the pic. With Lightroom the only way to get an edited image to iPhoto is to Export from LR and then import to iPhoto. (There was a plug-in to automate that process but I have no idea if it's been updated since LR 1.)
Aperture can share it's Previews with iPhoto, but that's all. Otherwise you need to do the export/import dance.
What communication there is between Aperture and iPhoto is designed to facilitate upgrading from iPhoto to Aperture. Yes, Aperture is a complete replacement for iPhoto.
Neither the iPhoto nor Aperture Libraries can live on your NAS. However, the file management tools in Aperture are such that you can easily store the files on your NAS while the Library is on the HD. You can also do this with iPhoto but I wouldn't recommend it.
Frankly, if you're a Raw shooter I don't understand why you would consider changing from the Pro level LR to a hiome user's iPhoto.
Regards
TD

What kind of file is "Quicktime Movie" file?

Hi, I posted this in the "Connecting the Fifth Generation iPod to Windows" section but I got no replys.
The Question was:
I was looking at my collection and I see some movies/videos with the extention "Quicktime movie file" what type of file is this?
My guess is that it is H.264 but please let me know if it isn't.
Thanks
Jason

"The QuickTime file format has been used as the basis of the MPEG-4 standard, developed by the International Organization for Standardization (ISO)."
http://developer.apple.com/documentation/QuickTime/QTFF/QTFFPreface/chapter1_section1.html

What type of file is "Quicktime movie file"?

Hello,
I was looking at my collection and I see some movies/videos with the extention "Quicktime movie file" what type of file is this?
My guess is that it is H.624 but please confirm that.
Thanks
Jason

"The QuickTime file format has been used as the basis of the MPEG-4 standard, developed by the International Organization for Standardization (ISO)."
http://developer.apple.com/documentation/QuickTime/QTFF/QTFFPreface/chapter1_section1.html

What is xtat file

Dear gurus
I am working on file adapter .my client told that he will send xtat file what is xtat file in file adapter .
Thanks in advance
a.kumar

Hi,
It is just format of the file similar to .doc,.pdf etc.
As a PI consultant you only need to bother about below
1)Is it just pick and drop or transformation required ?
2)If it requires transformation then need to focus on whether it is in xml format or not ,if not how to convert using fcc or custom module or java mapping .
3)Are they placing in file server or not ?
4)Does ports to your PI machine or not?
5)What service they enabled to PI (telnet,ssl,...etc)
Regards
Venkat

Today, I randomly happened to have less than 1GB of hard drive space left. I found very large "frame" files, what are they?

I found very large "frame" files, what are they & can I delete them? (See screenshot). I'm a (17 today)-year-old film-maker and can't edit in FCP X anymore because I "don't have enough space". Every time I try to delete one, another identical file creates itself...
If that can help: I just upgraded to FCP 10.0.4 and every time I launch it it asks to convert my current projects (I know it would do it at least once) and I accept, but everytime I have to get it done AGAIN. My computer is slower than ever and I have a deadline this friday
I also just upgraded to Mac OS X 10.7.4, and the problem hasn't been here for long, so it may be linked...
Please help me!
Alex

The first thing you should do is to back up your personal data. It is possible that your hard drive is failing. If you are using Time Machine, that part is already done.
Then, I think it would be easiest to reformat the drive and restore. If you ARE using Time Machine, you can start up from your Leopard installation disc. At the first Installer screen, go up to the menu bar, and from the Utilities menu, first select to run Disk Utility. Completely erase the internal drive using the Erase tab; make sure you have the internal DRIVE (not the volume) selected in the sidebar, and make sure you are NOT erasing your Time Machine drive by mistake. After erasing, quit Disk Utility, and select the command to restore from backup from the same Utilities menu. Using that Time Machine volume restore utility, you can restore it to a time and date immediately before you went on vacation, when things were working.
If you are not using Time Machine, you can erase and reinstall the OS (after you have backed up your personal data). After restarting from the new installation and installing all the updates using Software Update, you can restore your personal data from the backup you just made.

Best way to store a lot of image files??

I am working on a new project which involve user upload image to server and then do searching and view image. there will be a lot of image files. image will be displayed in web page as thumbnails, so there may be resized pic stored in server or resize on the fly.
What is the best way to store image like this? as files in file system? database? etc...?
Thank you very much

Since seraching is invloved use a database. A file system is not good place to search in.
Using a database will allow you to store additional data as well such as comments about the images or the ratings given by the users.
Of course you will have to develop a propper table structure for this.
Also do not generate thumbnails all the time instead cache them once they are created. Atleast the thumbnails of the images that are displayed often.
Also store the last updted dates with the images so you can loow the images to be cached by the clients or the proxy server. That will reduce repatitive requests for the same image from same client.

What is the optimum core configuration for a new Mac Pro to process and manipulate very large (80 megapixel) images using PhotoShop and Camera Raw?

Hello:
I will be using creative techniques to process and manipulate a large number (hundreds) of very large (80 megapixel) images captured using a medium format digital back (Phase One IQ180).
Final output will be digital fine art imagery printed using an Epson 11880 at large sizes (up to 60 inches x ?), retaining the highest possible quality and resolution. I will be using Adobe CC PhotoShop and Camera RAW as well as Capture One software. PhotoShop filters will be used extensively.
The Mac Pro needs to be optimized for the above purpose and be useful for at least five years. I plan to max out all the other options (RAM, graphics cards, storage). Performance is more important than cost.
The few discussions I have found that mention optimum core configurations seem to lean toward 6 or 8 (but likely are not taking into consideration my need for manipulating a large number of very large files), so I am looking to this foum for opinions.
Thank you,
Kent

See if this helps
http://macperformanceguide.com/index_topics.html#MacPro2013

Ways to handle large volume data (file size = 60MB) in PI 7.0 file to file

Hi,
In a file to file scenario (flat file to xml file), the flat file is getting picked up by FCC and then send to XI. In xi its performing message mapping and then XSL transformation in a sequence.
The scenario is working fine for small files (size upto 5MB) but when the input flat file size is more then 60 MB, then XI is showing lots of problem like (1) JCo call error or (2) some times even XI is stoped and we have to strat it manually again to function properly.
Please suggest some way to handle large volume (file size upto 60MB) in PI 7.0 file to file scenario.
Best Regards,
Madan Agrawal.

Hi Madan,
If every record of your source file was processed in a target system, maybe you could split your source file into several messages by setting up this in Recordset Per Messages parameter.
However, you just want to convert you .txt file into a .xml file. So, try firstly to setting up
EO_MSG_SIZE_LIMIT parameter in SXMB_ADM.
However this could solve the problem in Inegration Engine, but the problem will persit in Adapter Engine, I mean, JCo call error ...
Take into account that file is first proccessed in Adapter Engine, File Content Conversion and so on...
and then it is sent to the pipeline in Integration Engine.
Carlos

Yosemite is awful- unable to empty trash, volume options are haywire, error codes on file management, and files won't replace when moved. What are my options at this point?

Like the topic - "Yosemite is awful- unable to empty trash, volume options are haywire, error codes on file management, and files won't replace when moved. What are my options at this point?"
I installed this OS, and it looks nice, but it's terrible to use. I randomly have the volume controls get disabled, and sometimes the volume menu in the system tray does nothing when adjusted which I've never seen in an OS before. The trash won't empty. When I try to drag and drop newer files over existing ones, nothing happens. The old ones just stay there. I have to delete the old ones and then move the new ones. And when I go into the protected files to manage audio plugins (I make music), the first thing I hit are error codes galore, and password prompts that either don't popup when they should, or don't execute the command after I provide my password.
What can I do now that I installed the flames of **** onto my computer?

Back up all data before proceeding.
This procedure will unlock all your user files (not system files) and reset their ownership, permissions, and access controls to the default. If you've intentionally set special values for those attributes on any of your files, they will be reverted. In that case, either stop here, or be prepared to recreate the settings if necessary. Do so only after verifying that those settings didn't cause the problem. If none of this is meaningful to you, you don't need to worry about it, but you do need to follow the instructions below.
Step 1
If you have more than one user, and the one in question is not an administrator, then go to Step 2.
Triple-click anywhere in the following line on this page to select it:
sudo find ~ $TMPDIR.. -exec chflags -h nouchg,nouappnd,noschg,nosappnd {} + -exec chown -h $UID {} + -exec chmod +rw {} + -exec chmod -h -N {} + -type d -exec chmod -h +x {} + 2>&-
Copy the selected text to the Clipboard by pressing the key combination command-C.
Launch the built-in Terminal application in any of the following ways:
☞ Enter the first few letters of its name into a Spotlight search. Select it in the results (it should be at the top.)
☞ In the Finder, select Go ▹ Utilities from the menu bar, or press the key combination shift-command-U. The application is in the folder that opens.
☞ Open LaunchPad. Click Utilities, then Terminal in the icon grid.
Paste into the Terminal window by pressing command-V. I've tested these instructions only with the Safari web browser. If you use another browser, you may have to press the return key after pasting.
You'll be prompted for your login password, which won't be displayed when you type it. Type carefully and then press return. You may get a one-time warning to be careful. If you don’t have a login password, you’ll need to set one before you can run the command. If you see a message that your username "is not in the sudoers file," then you're not logged in as an administrator.
The command may take several minutes to run, depending on how many files you have. Wait for a new line ending in a dollar sign ($) to appear, then quit Terminal.
Step 2 (optional)
Take this step only if you have trouble with Step 1, if you prefer not to take it, or if it doesn't solve the problem.
Start up in Recovery mode. When the OS X Utilities screen appears, select
Utilities ▹ Terminal
from the menu bar. A Terminal window will open. In that window, type this:
resetp
Press the tab key. The partial command you typed will automatically be completed to this:
resetpassword
Press return. A Reset Password window will open. You’re not going to reset a password.
Select your startup volume ("Macintosh HD," unless you gave it a different name) if not already selected.
Select your username from the menu labeled Select the user account if not already selected.
Under Reset Home Directory Permissions and ACLs, click the Reset button.
Select
 ▹ Restart
from the menu bar.

Ok to store large files on desktop?

Ok to store large files on desktop?
Are there consequences (running slow) when storing a large files or numerous files on the desktop?
Sometimes I use the desktop to stores photos until I complete a project. There are other files on the desktop that I could remove if it causes the computer to slow down. If not, I'll keep them there because it's easy and convenient.
I have 4G of memory.

Hi Junk,
I can't think of any consequences to storing your large files on your desktop. After all, from your system's point of view, your desktop is just another folder. However, there is some system consequence to storing lots of files there, as it will decrease performance. But with 4GB of memory, I'm not sure how much of a difference you might see. And there's a big gap between "I'm storing 15 photos on my desktop" and "I can no longer see my hard disk icon in this mess." If the idea of even potential system slowdowns bothers you, create a work folder and leave it on your desktop to drop all of those photos in. If you're not getting too insane with the amount, though, I'm sure you'll be fine.
Hope that helps!
—Hazy

Disk Inspector reports that 260 GB of files are 'hidden.' What are these files and why is this so large??

I have iMac with 1 Terabyte storage. I just ran "Disk Inspector" and I have 260GB of "Hidden Items."
Have no clue what this is and why this is so large. Any help is appreciated.

They are required system files which are normally hidden to prevent accidental deletion.

I'm trying to play MIDI files in Quicktime. It works buy the volume is very low, even with both Quicktime and computer volume on max. What can I do?

I'm trying to play MIDI files in Quicktime. It works buy the volume is very low, even with both Quicktime and computer volume on max. What can I do?

I'm having the same issue. After opening Audio Midi Setup in Utilities I noticed that the master volume slider of the Built-In Output is very low. I am not able to slide to get any more level out of the system.
Can you adjust the master volume on your system?

If a pdf file is to large or complex to convert ,what do i do ?

if a pdf file is to large or complex to convert ,what is my next step?@

Hello,
an interesting instruction provides exporting pdf file per Indesign(!), Please have a look at:
https://helpx.adobe.com/indesign/using/exporting-publishing-pdf.html >>> Reducing the size of PDFs
Hans-Günter

In FINDER, the IMAGES section of 'All My FIles', what is: myDocuments.cannedSearch? Looks like images of jpegs & PDFs I viewed, even if I had deleted it.

I previously posted a similar question, but I was not clear in my message. I know that files (categorized as PDF Documents, Movies, Presentations, Spreadsheets, etc.) in "all my files' in the FINDER are all my files and if I delete them, I am deleting the original copies. However, the IMAGES category has thousands of images of PDF and jpeg files, even ones I had deleted. I do not have thousands of pics or pdf documents. It looks like these images are files I had opened, even if I deleted the files. I want to clean up files on MAC. Any suggestions? See info below regarding IMAGES category of ALL MY FILES. When I clic on INFO, it says it is a saved query.

I actually just noticed the "media" option is no longer listed along the left side of Finder. I think that is where I accessed iphoto and my other picture folder.
There is no media option in the Finder. That only turns up in Open... Dialogues like this:

Store large volume of Image files, what is better ? File System or Oracle

Similar Messages

Maybe you are looking for