Working with large data and PHP

Using a backend MySQL database, I'd like to interact with
this data using PHP. I got information from one link:
http://www.sephiroth.it/tutorials/flashPHP/pageable_recordset/
Which is exactly what I need. Page the Mysql server every
time you need to grab data and also have a listener that will
automatically update the client. However I can't seem to get it
work. Does anyone know how to make this happen using a repeater and
a panel?

Read Developper's Guide at
http://www.adobe.com/support/documentation/en/flex/.
You'll find everything there !

Similar Messages

Bookmark sync not working with large set and maybe javascript bookmark bookmarklets??

Bookmark sync via iCloud doesn't seem to work for me. I use Chrome as my primary browser but keep a huge set of bookmarks synced to Safari via Xmarks.
Basically I either get no bookmarks on my iOS devices (iPhone 4 and iPad 1) or a huge number of repeated versions of 1Password's very large java bookmark.
I've tested by deleting all my Safari bookmarks and starting with a clean Mac. When I do this everything works and changes on any device propagate to the others.
As soon as I bring in my big bookmark set from Xmarks, it all breaks.
Do I need to perform the bookmark clean up I've been putting off for a decade or so?

Can't help with your bidirectional bookmark syncing question but try this for backing up your touch's bookmarks.
Export/backup all of your Safari/Internet Explorer bookmarks (depending on where you're syncing them from) and keep them is a safe place. This is your master bookmarks list. Then delete all bookmarks/folders from your browser. Now sync your bookmarks with the touch with the following iTunes settings:
Under your touch's 'Info' tab, select the checkbox 'Sync bookmarks with Internet Explorer/Safari'.
After syncing, your touch's bookmarks (and only those from your touch) will be synced to your browser. Now export/backup these bookmarks...these are your touch's bookmarks. Now you have a both a backup of your master bookmarks and your touch's bookmarks. When you want to sync/restore your touch's bookmarks back to the touch, delete all your PC browser's bookmarks and import the touch bookmarks:
Under iTunes/your touch/'Info'/Web browser & Advanaced, select the checkboxes 'sync bookmarks with IE/Safari' and 'Replace information on this iPod: Bookmarks' and sync again.
This will transfer your touch's bookmarks back to the touch. Now delete all bookmarks again from IE/Safari and re-import your master bookmarks. Note that after importing them back in, they will likely be in alphabetical order rather than the date added order.
Also note that (at least in Windows) your browser needs to be open for bookmark syncing to take place.

Working with large data

I have to create a database with specific distribution of key size and data
size. Key size is some few bytes, and data size varies in a great range from
few bytes to some megabytes with average size near 64K. Overall size of a
database filled by key/data pairs is some gigabytes. One can imagine our
key/data pairs form context index (inverted file) for a large set of
Russian/English texts.
Could you recommend the best way to configure such a data storage? I mean the
best random read speed for key/data pairs in a given database and good enough
write speed (for "context index" updating).

Hi,
The most important configuration item in your case will be the cache size configuration. The larger the cache the better performance will be - especially for random read-oriented usage. Documentation about the cache configuration API are here:
http://www.sleepycat.com/docs/api_c/db_set_cachesize.html
An article about tuning cache size is available here:
http://www.sleepycat.com/newsletters/0511/a31_Perf_Size.html
Selecting the format for the database will also have an impact. Given your description I suggest that hash is likely the best solution - since the data access will be random. You should test with both hash and btree. An article describing the benefits/drawbacks of both is here:
http://www.sleepycat.com/docs/ref/am_conf/select.html
Then you might want to adjust the pagesize - given that your data items are generally large, a bigger page size will probably result in better performance. API here:
http://www.sleepycat.com/docs/api_c/db_set_pagesize.html
The db_stat utility is a very useful tool for tuning your database. Documentation can be found here:
http://www.sleepycat.com/docs/utility/db_stat.html
If you have any specific questions I will be glad to help.
Regards,
Alex

Working with midi data and automation data, assigning parameters

I have a midi file with CC data in it. Controller 7 (volume) Controller 1 (mod) On page 600 the manual explains converting midi controller data to automation and vice-versa. I don't see any info on how to manage what CC data is mapped to what automation though. When I convert this midi file, I just get volume automation. Volume automation is just automatically there when you display automation, so I'm guessing that might have something to do with it. The problem is, I have CC1 data (and will be bringing in midi files that have data on many other CC#'s in the future) and I don't see how to map it to the parameters of plug-ins. I'd like to bring in midi files with information on any CC# I wanted, and then map that to any parameter of any plug-in I wanted. I couldn't figure out how to do this with the automation conversion process though.
This issue led me to another question (sorry). Since I wasn't getting this to work I decided to try something else. I thought I'd try opening the controller assignments (command K) window and just assigning the CC data to control parameters that way. So I use my midi hardware controller to set a parameter to be controlled by CC#1. This works fine, and I can control the parameter with my midi controller. When I press play however, hoping that the CC#1 data in the midi file I imported will control this parameter, it doesn't seem to do it.
If someone to help me I would appreciate it very much, I wouldn't be surprised if I'm just missing something in one of the manuals, but I have not worked this way with Logic very much, I've just always used it's automation system and haven't ventured into dealing with CC data. Sorry for the long post, I just wanted to explain clearly (hope I have)

Balie Todd wrote:
I'd like to bring in midi files with information on any CC# I wanted, and then map that to any parameter of any plug-in I wanted. I couldn't figure out how to do this with the automation conversion process though.
First you cross too many questions, that's why I will try to answer one by one when have time cause they need some graphics etc.
As a beginnings have a look at this forum [LINK|http://discussions.apple.com/thread.jspa?messageID=10612612&#10612612], where I show several scenarios how to manage CC7 and CC10 in the environment. It is real time processing. You can do some post processing using the Transform Window as well. Hope to have time to show you some Transform Mappers.
In addition it is possible to Map any CC# to any Plugin parameter in the Environment using transformer Mapper object ( realtime ), or using the Transform Window as well. ( If I have time later this evening will make same examples...
So I use my midi hardware controller to set a parameter to be controlled by CC#1. This works fine, and I can control the parameter with my midi controller. When I press play however, hoping that the CC#1 data in the midi file I imported will control this parameter, it doesn't seem to do it.
This is possible as well but using some Environment tricks with IAC port, a dummy "Learn" fader and probably condition transformer ( I have show many times such schemes but do not have time to search right now - hope to have time to make new one according your scenario ).
Will come back soon with more info and schemes...
Regards
!http://img59.imageshack.us/img59/4967/aglogo45.gif! [www.audiogrocery.com|http://www.audiogrocery.com]

Working with Large data sets Waveforms

When collection data at a high rate ( 30K ) and for a long period (120 seconds) I'm unable rearrange the data due to memory errors, is there a more efficient method?
Attachments:
Convert2Dto1D.vi ‏36 KB

Some suggestions:
Preallocate your final data before you start your calculations. The build array you have in your loop will tend to fragment memory, giving you issues.
Use the In Place Element to get data to/from your waveforms. You can use it to get single waveforms from your 2D array and Y data from a waveform.
Do not use the Transpose and autoindex. It is adding a copy of data.
Use the Array palette functions (e.g. Reshape Array) to change sizes of current data in place (if possible).
You may want to read Managing Large Data Sets in LabVIEW.
Your initial post is missing some information. How many channels are you acquiring and what is the bit depth of each channel? 30kHz is a relatively slow acquisition rate for a single channel (NI sells instruments which acquire at 2GHz). 120s of data from said single channel is modestly large, but not huge. If you have 100 channels, things change. If you are acquiring them at 32-bit resolution, things change (although not as much). Please post these parameters and we can help more.
This account is no longer active. Contact ShadesOfGray for current posts and information.

Working with Connect Pro and PHP

Hi,
I understand that I can use PHP's POST and GET functions to start a Connect Pro Session. Can anyone tell me if there are other fuctions that can be incorporated within a PHP site? I would like a recorded video of each session to be automaily saved to my sight and associated with all profiles of the users who were involved with the session.
Thanks,
John

Read Developper's Guide at
http://www.adobe.com/support/documentation/en/flex/.
You'll find everything there !

Working with Large List in sharepoint 2010

Hi All
I have a list with almost 10k records in my sharepoint list and based on some business requirement i am binding (almost 6k records) the data to asp.net grid view and this will visible on the home page of the portal where most of the users will access. Can
someone please guide the best method to reduce the performance inorder the program to hit the SP list everytime the page loads...
Thanks & Regards
Rakesh Kumar

Hi,
If you are Working with large data retrieval from the content database (SharePoint list), the points below for your reference:
1. Limit the number of returned items.
SPQuery query = new SPQuery();
query.RowLimit =6000; // we want to retrieve2000 items
query.ListItemCollectionPosition = prevItems.ListItemCollectionPosition; // starting at a previous position
SPListItemCollection items = SPContext.Current.List.GetItems(query);
2. Limit the number of returned columns.
SPQuery query = new SPQuery();
query.ViewFields = "";
3. Query specific items using CAML (Collaborative Markup Language).
SPQuery query = new SPQuery();
query.Query = "15";
4.Use ContentIterator class
https://spcounselor-public.sharepoint.com/Blog/Post/2/Querying-a--big-list--with-ContentIterator-and-multiple-filters
5. Create a Stored Procedure in Database to get the special data, create a web service to get the data, when create a web part to show the data in home page.
Best Regards
Dennis Guo
TechNet Community Support

Working with Large Numbers

Hi there,
I am currently doing a school assignment and not looking for answers but just a little guidance.
I am working with large numbers and the modulo operator.
I might have some numbers such as :
int n = 221;
int e = 5;
int d = 77;
int message = 84;
int en = (int) (Math.pow(message, e) % n);
int dn = (int) (Math.pow(en, d) % n);Would there be a better way to do this kind of calculation. The dn value should come out the same as message. But I always get something different and I think I might be losing something in the fact that an int can only hold smaller values.

EJP wrote:
It might make sense in some contexts to have a positive and negative infinity.
Yes, perhaps that's a better name. Guess I was harking back to old COBOL days :-).(*)
But the reason these things exist in FP is because the hardware can actually deliver them. That rationale doesn't apply to BIgInteger.Actually, it does. All I'm talking about is a value that compares higher or lower than any other. That could be done either by a special internal sign value (my slight preference) or by simply adding code to compareTo(), equals() and hashCode() methods that takes the two constants into account (as they already do with ZERO and ONE).
Don't worry, I'm not holding my breath; but I have come across a few situations in which values like that would have been useful.
Winston
Edited by: YoungWinston on Mar 22, 2011 9:07 AM
(*) Actually, '±infinity' tends to suggest a valid arithmetic value, and I wasn't thinking of changing existing BigInteger/BigDecimal maths (except perhaps to throw an exception if either value is involved).

How to improve performance(insert,delete and search) of table with large data.

Hi,
I am having a table which is used for maintaining history and have a large data and that keeps on increasing or decreasing based on the business rules.
I am getting performance issues with this table which searching for any records or while inserting new data into it. I have already used index in this table but still I am facing lot of issues related to performance.
Also, we used to insert bulk data into this table.
Can we have any solution to achieve this, any solutions are greatly appreciated.
Thanks in Advance!

Please do not duplicate your posts across forums. It's considered bad practice and rude, as people will not know what answers you've already received and may end up duplicating the effort.
Locking this thread - answer on other thread please

Working with AVCHD: Log and Transfer FAQs

Disclaimer: Apple does not necessarily endorse any suggestions, solutions, or third-party software products that may be mentioned in the topic below. Apple encourages you to first seek a solution at Apple Support. The following links are provided as is, with no guarantee of the effectiveness or reliability of the information. Apple does not guarantee that these links will be maintained or functional at any given time. Use the information below at your own discretion.
Working with AVCHD: Log and Transfer FAQs
Log and Transfer was introduced to the product chain in Final Cut Express 4, and is the program's process for ingesting files in the AVCHD format utilizing MPEG-4 AVC/H.264 video coding and either Dolby AC-3 or uncompressed linear PCM audio coding. The most frequently observed problems relating to the Log and Transfer process are generally quite easily addressed.
What should I know before attempting to ingest my footage?
The following facts regarding the Log and Transfer process were adapted from the [Apple Knowledge Base article|http://support.apple.com/kb/TA24840?viewlocale=en_US] on transferring AVCHD in Final Cut Pro 6.0.1 but still hold true for the current version of FCE:
AVCHD support is available only on Intel-based Macintosh computers.
AVCHD camcorders typically connect to computers via USB 2.0, not FireWire.
DVD-based AVCHD camcorders are not supported in Mac OS X 10.4 or earlier.
Standard definition video recorded with AVCHD camcorders cannot be accessed in the Log and Transfer window. SD and HD clips should not be mixed on the same HDD.
Preview of AVCHD video in the Log and Transfer window is limited to 1x forward playback. Scrubbing the playhead does not update the video in the Preview area until you stop scrubbing.
When you choose to transfer AVCHD audio in the Logging area, audio is automatically mixed down to stereo.
AVCHD footage is not ingested natively but is transcoded to the Apple Intermediate Codec.
Apple Intermediate Codec files are significantly larger than the original AVCHD files. FCE does not estimate the size of a transcoded AVCHD file before ingesting, so it is possible to exceed the available space on your capture disk during transfer. Before ingesting, keep in mind that the disk space required is about ten times greater than the size of the native AVCHD file.
Which specific formats can be ingested by Log and Transfer?
Only AVCHD footage can be ingested through the Log and Transfer process. Other formats and variants such as AVCHD Lite are not supported. All video should be recorded in a frame size of either 1440x1080 or 1920x1080 and a frame rate of 25 or 29.97 frames per second. Audio recorded on AVCHD camcorders is generally at a rate of 48kHz. If your camera shoots in a format that does not comply with one of these setups then it is most likely not supported by the application and is not guaranteed to work correctly.
Which Easy Setup preset should I select before ingesting my footage?
It is extremely important to select the preset that matches your media before you attempt to ingest via Log and Transfer. There is only one Easy Setup preset that matches the frame size and rate of your footage. If you intend to ingest AVCHD clips through Log and Transfer you will first need to select one of the following presets:
AVCHD-Apple Intermediate Codec 1440x1080i50
AVCHD-Apple Intermediate Codec 1440x1080i60
AVCHD-Apple Intermediate Codec 1920x1080i50
AVCHD-Apple Intermediate Codec 1920x1080i60
If these four options do not appear under your Easy Setup list then you probably need to run Software Update and update to FCE 4.0.1. DO NOT cycle through the available presets until you find one that works - there is only one preset that matches your media and it should be evident which one to choose by the way it is named. Select the AVCHD preset that mentions your frame size and frame rate (i50 presets support both 50i and 25p frame rates; similarly, i60 presets support both 60i and 29.97p frame rates). If you do cycle through many options in the hope of selecting the correct one through guess-and-check you will probably have corrupted your preference file and need to [trash your preferences|http://support.apple.com/kb/TS2611] before continuing.
Why don't my clips appear when I open the Log and Transfer window?
Make sure your camcorder records AVCHD in a format supported by FCE4. It should be connected to one of your computer's USB ports (never connect it to an external hard drive or other external device such as your keyboard or a USB hub), powered on, and set to video playback mode.
If the camcorder +does not+ mount on your Desktop your USB port or cable may be malfunctioning. Try plugging it in to another port or using a different USB cable to see if it makes a difference.
If the camcorder does mount on your Desktop but the clips do not appear in the Log and Transfer window you will need to point the program to the directory that contains your media.
First, click the Add Folder button in the upper left of the Log and Transfer window. The button looks like this:
Then, you will need to navigate to the root directory in which your media is stored. Most camcorders call this folder "PRIVATE" or "AVCHD" and it should be located at the very top level of the camcorder when mounted. If your camcorder records to a valid AVCHD directory structure then your clips should load after the Add Folder process is completed.
I would like to keep a working backup of my original clips so that I can delete them off the camcorder's internal memory. How should I do this?
There are two main ways to keep a backup of your clips. The first is to ingest every clip on your camcorder through the Log and Transfer process, and store the Apple Intermediate Codec files the application creates in an easily accessible place such as an external FireWire drive. The advantage of this method is that you have the ability to open back up old editing projects, reconnect the media as necessary, and keep going as if you'd never left. However, due to the large amount of space AIC takes up (usually somewhere in the vicinity of 41-50GB an hour when transcoded from AVCHD), this is not always feasible.
Another often preferable approach is to copy over the root directory in which your clips are stored (usually named "PRIVATE" or "AVCHD") from the camcorder to your computer or an external drive. As long as you maintain the original file structure, you will be able to ingest your clips again at a later date using the "Add Folder" process detailed above to navigate to their new location.
My clips appear, but when I attempt to ingest a red exclamation point appears and the window displays the message "Error: No Data"
This is one of the most common problems that users experience when trying to ingest their media through the Log and Transfer process. Luckily, it also has one of the easiest solutions.
In the Log and Transfer window, click on the gear icon in the top right and select "Preferences" in the contextual menu that appears.
Make sure the video is set to transcode to the Apple Intermediate Codec and the audio is set to Plain Stereo.
After doing so, hit OK and try to ingest again.
I am experiencing a problem that is not covered in this tip or the solutions here do not work.
The best thing to do at this point would be to trash your preferences and start a new topic in the [FCE Forum|http://discussions.apple.com/forum.jspa?forumID=936&start=0] with a detailed description of your problem. Check out the user tip on [Basic Troubleshooting in Final Cut Express|http://discussions.apple.com/thread.jspa?threadID=2662419] for more information.
This is the 1st version of this tip. It was submitted on December 03, 2010 by skalicki`.
Do you want to provide feedback on this User Contributed Tip or contribute your own? If you have achieved Level 2 status, visit the User Tips Library Contributions forum for more information.

Disclaimer: Apple does not necessarily endorse any suggestions, solutions, or third-party software products that may be mentioned in the topic below. Apple encourages you to first seek a solution at Apple Support. The following links are provided as is, with no guarantee of the effectiveness or reliability of the information. Apple does not guarantee that these links will be maintained or functional at any given time. Use the information below at your own discretion.
Working with AVCHD: Log and Transfer FAQs
Log and Transfer was introduced to the product chain in Final Cut Express 4, and is the program's process for ingesting files in the AVCHD format utilizing MPEG-4 AVC/H.264 video coding and either Dolby AC-3 or uncompressed linear PCM audio coding. The most frequently observed problems relating to the Log and Transfer process are generally quite easily addressed.
What should I know before attempting to ingest my footage?
The following facts regarding the Log and Transfer process were adapted from the [Apple Knowledge Base article|http://support.apple.com/kb/TA24840?viewlocale=en_US] on transferring AVCHD in Final Cut Pro 6.0.1 but still hold true for the current version of FCE:
AVCHD support is available only on Intel-based Macintosh computers.
AVCHD camcorders typically connect to computers via USB 2.0, not FireWire.
DVD-based AVCHD camcorders are not supported in Mac OS X 10.4 or earlier.
Standard definition video recorded with AVCHD camcorders cannot be accessed in the Log and Transfer window. SD and HD clips should not be mixed on the same HDD.
Preview of AVCHD video in the Log and Transfer window is limited to 1x forward playback. Scrubbing the playhead does not update the video in the Preview area until you stop scrubbing.
When you choose to transfer AVCHD audio in the Logging area, audio is automatically mixed down to stereo.
AVCHD footage is not ingested natively but is transcoded to the Apple Intermediate Codec.
Apple Intermediate Codec files are significantly larger than the original AVCHD files. FCE does not estimate the size of a transcoded AVCHD file before ingesting, so it is possible to exceed the available space on your capture disk during transfer. Before ingesting, keep in mind that the disk space required is about ten times greater than the size of the native AVCHD file.
Which specific formats can be ingested by Log and Transfer?
Only AVCHD footage can be ingested through the Log and Transfer process. Other formats and variants such as AVCHD Lite are not supported. All video should be recorded in a frame size of either 1440x1080 or 1920x1080 and a frame rate of 25 or 29.97 frames per second. Audio recorded on AVCHD camcorders is generally at a rate of 48kHz. If your camera shoots in a format that does not comply with one of these setups then it is most likely not supported by the application and is not guaranteed to work correctly.
Which Easy Setup preset should I select before ingesting my footage?
It is extremely important to select the preset that matches your media before you attempt to ingest via Log and Transfer. There is only one Easy Setup preset that matches the frame size and rate of your footage. If you intend to ingest AVCHD clips through Log and Transfer you will first need to select one of the following presets:
AVCHD-Apple Intermediate Codec 1440x1080i50
AVCHD-Apple Intermediate Codec 1440x1080i60
AVCHD-Apple Intermediate Codec 1920x1080i50
AVCHD-Apple Intermediate Codec 1920x1080i60
If these four options do not appear under your Easy Setup list then you probably need to run Software Update and update to FCE 4.0.1. DO NOT cycle through the available presets until you find one that works - there is only one preset that matches your media and it should be evident which one to choose by the way it is named. Select the AVCHD preset that mentions your frame size and frame rate (i50 presets support both 50i and 25p frame rates; similarly, i60 presets support both 60i and 29.97p frame rates). If you do cycle through many options in the hope of selecting the correct one through guess-and-check you will probably have corrupted your preference file and need to [trash your preferences|http://support.apple.com/kb/TS2611] before continuing.
Why don't my clips appear when I open the Log and Transfer window?
Make sure your camcorder records AVCHD in a format supported by FCE4. It should be connected to one of your computer's USB ports (never connect it to an external hard drive or other external device such as your keyboard or a USB hub), powered on, and set to video playback mode.
If the camcorder +does not+ mount on your Desktop your USB port or cable may be malfunctioning. Try plugging it in to another port or using a different USB cable to see if it makes a difference.
If the camcorder does mount on your Desktop but the clips do not appear in the Log and Transfer window you will need to point the program to the directory that contains your media.
First, click the Add Folder button in the upper left of the Log and Transfer window. The button looks like this:
Then, you will need to navigate to the root directory in which your media is stored. Most camcorders call this folder "PRIVATE" or "AVCHD" and it should be located at the very top level of the camcorder when mounted. If your camcorder records to a valid AVCHD directory structure then your clips should load after the Add Folder process is completed.
I would like to keep a working backup of my original clips so that I can delete them off the camcorder's internal memory. How should I do this?
There are two main ways to keep a backup of your clips. The first is to ingest every clip on your camcorder through the Log and Transfer process, and store the Apple Intermediate Codec files the application creates in an easily accessible place such as an external FireWire drive. The advantage of this method is that you have the ability to open back up old editing projects, reconnect the media as necessary, and keep going as if you'd never left. However, due to the large amount of space AIC takes up (usually somewhere in the vicinity of 41-50GB an hour when transcoded from AVCHD), this is not always feasible.
Another often preferable approach is to copy over the root directory in which your clips are stored (usually named "PRIVATE" or "AVCHD") from the camcorder to your computer or an external drive. As long as you maintain the original file structure, you will be able to ingest your clips again at a later date using the "Add Folder" process detailed above to navigate to their new location.
My clips appear, but when I attempt to ingest a red exclamation point appears and the window displays the message "Error: No Data"
This is one of the most common problems that users experience when trying to ingest their media through the Log and Transfer process. Luckily, it also has one of the easiest solutions.
In the Log and Transfer window, click on the gear icon in the top right and select "Preferences" in the contextual menu that appears.
Make sure the video is set to transcode to the Apple Intermediate Codec and the audio is set to Plain Stereo.
After doing so, hit OK and try to ingest again.
I am experiencing a problem that is not covered in this tip or the solutions here do not work.
The best thing to do at this point would be to trash your preferences and start a new topic in the [FCE Forum|http://discussions.apple.com/forum.jspa?forumID=936&start=0] with a detailed description of your problem. Check out the user tip on [Basic Troubleshooting in Final Cut Express|http://discussions.apple.com/thread.jspa?threadID=2662419] for more information.
This is the 1st version of this tip. It was submitted on December 03, 2010 by skalicki`.
Do you want to provide feedback on this User Contributed Tip or contribute your own? If you have achieved Level 2 status, visit the User Tips Library Contributions forum for more information.

Working with large Artboards/Files in Illustrator

Hello all!
I'm currently designing a full size film poster for a client. The dimensions of the poster are 27" x 40" (industry standard film poster).
I am a little uncertain in working with large files in Illustrator, so I have several problems that have come up in several design projects using similar large formats.
The file size is MASSIVE. This poster uses several large, high-res images that I've embedded. I didn't want them to pixelate, so I made sure they were high quality. After embedding all these images, along with the vector graphics, the entire .ai file is 500MB. How can I reduce this file size? Can I do something with the images to make the .ai file smaller?
I made my artboard 27" x 40" - the final size of the poster. Is this standard practice? Or when designing for a large print format, are you supposed to make a smaller, more manageable artboard size, and then scale up after to avoid these massive file sizes?
I need to upload my support files for the project, including .ai and .eps - so it won't work if they're 500MB. This would be good info to understand for all projects I think.
Any help with this would be appreciated. I can't seem to find any coherent information pertaining to this problem that seems to address my particular issues. Thank you very much!
Asher

Hi Asher,
It's probably those high-res images you've embedded. Firstly, be sure your images are only as large as you need them Secondly, a solution would be to use linked images while you're working instead of embedding them into the file.
Here is a link to a forum with a lot of great discussion about this issue, to get you started: http://www.cartotalk.com/lofiversion/index.php?t126.html
And another: http://www.graphicdesignforum.com/forum/archive/index.php/t-1907.html
Here is a great list of tips that someone in the above forum gave:
-Properly scale files. Do not take a 6x6' file then use the scaling tool to make it a 2x2' Instead scale it in photoshop to 2x2 and reimport it. Make a rule like over 20%, bring it back in to photoshop for rescaling.
-Check resolutions. 600dpi is not going to be necessary for such and such printer.
-Delete unused art. Sloppy artists may leave old unused images under another image. The old one is not being used but it still takes up space, therefore unecessarily inflating your file.
-Choose to link instead of embedd. This is your choice. Either way you still have to send a large file but many times linking is less total MB then embedding. Also embedding works well with duplicated images. That way multiple uses link to one original, whereas embedding would make copies.
-When you are done, using compression software like ZIP or SIT (stuffit)
http://www.maczipit.com/
Compression can reduce file sizes alot, depending on the files.
This business deals with alot of large files. Generally people use FTP's to send large files, or plain old CD. Another option is using segmented compression. Something like winRAR/macRAR or dropsegment (peice of stuffit deluxe) compresses files, then breaks it up into smaller manageble pieces. This way you can break up a 50mb file into say 10x 5mb pieces and send them 5mb at a time.
http://www.rarlab.com/download.htm
*make sure your client knows how to uncompress those files. You may want to link them to the site to download the software."
Good luck!

Running out of memory while using cursored stream with large data

We are following the suggestions/recommendations for the cursored stream:
CursoredStream cursor = null;
          try
               Session session = getTransaction();
               int batchSize = 50;
               ReadAllQuery raq = getQuery();
               raq.useCursoredStream(batchSize, batchSize);
               int num = 0;
               ArrayList<Request> limitRequests = null;
               int totalLimitRequest = 0;
               cursor = (CursoredStream) session.executeQuery(raq);
               while( !cursor.atEnd() )
                    Request request = (Request) cursor.read() ;
                    if( num == 0 )
                         limitRequests = new ArrayList<Request>(batchSize);
                    limitRequests.add(request);
                    totalLimitRequest++;
                    num++;
                    if( num >= batchSize )
                         log.warn("Migrating batch of " + batchSize + " Requests.");
                         updateLimitRequestFillPriceForBatch(limitRequests);
                         num = 0;
                         cursor.releasePrevious();
               if( num > 0 )
                    updateLimitRequestFillPriceForBatch(limitRequests);
               cursor.close();
We are committing every 50 records in the unit of work, if we set DontMaintianCache on the ReadAllQuery we are getting PrimaryKeyExceptions intermittently, and we do not see much difference in the IdentityMap size.
Any suggestions/ideas for dealing with large data sets? Thanks

Hi,
If I use read-only classes with CursoredStream and execute the query within UOW, should I be saving any memory?
I had to use UOW because when I use Session to execute the query I get
6115: ISOLATED_QUERY_EXECUTED_ON_SERVER_SESSION
Cause: An isolated query was executed on a server session: queries on isolated classes, or queries set to use exclusive connections, must not be executed on a ServerSession or in CMP outside of a transaction.
I assume marking the descriptor as read-only will avoid registering in UOW, but I want to make sure that this is the case while using CursoredStream.
We are running in OC4J(OAS10.1.3.4) with BeanManagedTransaction.
Please suggest.
Thanks
-Raam
Edited by: Raam on Apr 2, 2009 1:45 PM

Is anyone working with large datasets ( 200M) in LabVIEW?

I am working with external Bioinformatics databasesa and find the datasets to be quite large (2 files easily come out at 50M or more). Is anyone working with large datasets like these? What is your experience with performance?

Colby, it all depends on how much memory you have in your system. You could be okay doing all that with 1GB of memory, but you still have to take care to not make copies of your data in your program. That said, I would not be surprised if your code could be written so that it would work on a machine with much less ram by using efficient algorithms. I am not a statistician, but I know that the averages & standard deviations can be calculated using a few bytes (even on arbitrary length data sets). Can't the ANOVA be performed using the standard deviations and means (and other information like the degrees of freedom, etc.)? Potentially, you could calculate all the various bits that are necessary and do the F-test with that information, and not need to ever have the entire data set in memory at one time. The tricky part for your application may be getting the desired data at the necessary times from all those different sources. I am usually working with files on disk where I grab x samples at a time, perform the statistics, dump the samples and get the next set, repeat as necessary. I can calculate the average of an arbitrary length data set easily by only loading one sample at a time from disk (it's still more efficient to work in small batches because the disk I/O overhead builds up).
Let me use the calculation of the mean as an example (hopefully the notation makes sense): see the jpg. What this means in plain english is that the mean can be calculated solely as a function of the current data point, the previous mean, and the sample number. For instance, given the data set [1 2 3 4 5], sum it, and divide by 5, you get 3. Or take it a point at a time: the average of [1]=1, [2+1*1]/2=1.5, [3+1.5*2]/3=2, [4+2*3]/4=2.5, [5+2.5*4]/5=3. This second method required far more multiplications and divisions, but it only ever required remembering the previous mean and the sample number, in addition to the new data point. Using this technique, I can find the average of gigs of data without ever needing more than three doubles and an int32 in memory. A similar derivation can be done for the variance, but it's easier to look it up (I can provide it if you have trouble finding it). Also, I think this funtionality is built into the LabVIEW pt by pt statistics functions.
I think you can probably get the data you need from those db's through some carefully crafted queries, but it's hard to say more without knowing a lot more about your application.
Hope this helps!
Chris
Attachments:
Mean Derivation.JPG ‏20 KB

Adobe Dreamweaver CS3 with ASP coldfusion and PHP search?

I am working through Adobe Dreamweaver CS3 with ASP
coldfusion and PHP with the intent of creating a data base site
that primarily requires a search engine on the index page. The
search page in the book uses specific search criteria ensuring that
there will always be a match to the data base but I need to create
an input text field search engine. My Q, if there is no data within
the data base that match the variable, will it return with an error
requiring an argument in the code to respond to no matching data.
Hopefully there are comprehensive online recourses for these search
engines as they are so common. I am a complete newby to all aspects
of web development and code so it will need to be a thorough
explanation, an idiots guide. ANY HELP GREATLY APPRECIATED

Rob,
Let me try to answer your questions, understanding that my
answers are laced with my experience and therefore my bias.
1. For the most part the SQL that you write for Access, MS
SQL or MySQL is going to be the same. I won't make that claim about
Oracle or PostgreSQL because I have limited or no experience in
those areas. To be sure when you head down the road with any DBMS
there are finer points on 'value added' features (to over simplify
things like MS SQL's Transact SQL), but for the most part when you
are pulling out, inserting or deleting records the SQL that you
write will be the same. In fact, I don't think I had to make any of
the SQL in the book different to deal with the back end db.
2. As for your host steering you away from MySQL, I don't
know why that would be. It is MUCH more stable and robust than
Access for web development. My research showed that you can use
MySQL just fine with ASP though that clearly wouldn't be
Microsoft's first choice.
3. A quick aside, the fact that you are building ASP on a Mac
would seem to indicate that you are developing on your production
server. That's never a good idea. Go local if you can. If you have
a newer Intel Mac you can use VM Fusion, Parallels or even Boot
Camp to run Windows in a virtual environment and develop there,
only moving your code to production when you are happy with it.
There is of course the fact that they two environments would not be
identical and you may need to change a few references, but it is
still a better practice than developing on your live site. Some web
hosts give you the ability to set up a subdomain. If yours does,
you could set one up with a duplicate of your site and develop
there.
4. Let me try to pull your 'rant-let' apart into two issue. I
begin with a question. I'm not sure what the Mac part has to do
with it. The primary language I work in at my day job is
ColdFusion. The majority of the most serious developers I know in
the CF world work on Macs. I am forced into the PC world because of
a cultural bias at the business school where I work. Having said
that, the only time I have felt there was ANY difference in the
tools, resources, choices I had was in any way affected by platform
was when Adobe still had not released Flex Builder 2 for the Mac.
Now that is behind us as well. This reaction of mine may be based
on the fact that Access never comes into play in the work that I
do. I regularly work with MS SQL and MySQL. (OK, so SQL Enterprise
Manager, the MS tool of choice to talk to MS SQL is Windows only,
but I only use that at the day job. But even there you can find
cross-platform, free tools like Aqua Data Studio to talk from a Mac
to MS SQL.) If I'm missing your point on the Mac, please let me
know. Since the advent of the Intel Mac I think it is hands-down
the best choice for a web developer. You can have Mac, Windows and
Unix all on one machine. It doesn't get any better (providing you
have the RAM :-)).
As for the security question, it is somewhat analogous to the
war on viruses and spyware. As the defenses get better, so do the
attacks. Most languages have developed functions for vetting user
input variables and best practices for building these things. It is
mainly a user education issue. When I took over the book you are
working out of, the one thing Jeffrey told me it needed was more
security. I did my best to put more in under the constraints of the
publisher. They didn't want the book to be any longer than the
previous version. I did my best to modify the code used to include
"string safe" functions and to add verbiage about the importance of
security. It was clear that I could not add as much as was needed
and I said so in the book. Much more is needed for a serious
treatment of the subject and not acknowledging that would have been
a disservice to the readers. I've been working as a developer for
over a decade and I don't know all there is to know about it. It's
a constantly changing field, just like all of web development. In
fact next week I'm going through a 2-day SANS Web Application
Security workshop in hopes of learning more. So at the risk of
sounding like an industry apologist, I don't think it's a
shortcoming of the development community. It is, as they say, "a
developing situation".
HTH,
Bob
http://bobflynn.info/

The Essential Guide to Dreamweaver CS3 with CSS, Ajax, and PHP by David Powers

The Essential Guide to Dreamweaver CS3 with CSS, Ajax, and
PHP by David Powers:
Would this book help me learn about Spry and Ajax and how
applications can be made with them, if I my e-commerce site runs on
a compiled C code shopping cart (ShopSite) but supports PHP
server-side scripts or is it only for shopping carts based on PHP
itself (like X-Cart)?
I am not sure. Someone please let me know so I can purchase
this book. Thanks.

ahsenabro wrote:
> 1. Do I have to be a developer/programmer in order to
learn from this book or
> can a tech-savvy entrepreneur can also get some gems out
of it and direct his
> programmer?
You don't need any prior knowledge of Ajax or PHP, but you do
need to
know the basics of HTML and CSS. As I say in the
introduction, "You
don't need to be an expert, but you do need to have an
inquiring mind.
It doesn't teach the basics of web design, nor does it
attempt to list
every single feature in Dreamweaver CS3. There are plenty of
other books
to fill that gap. However, by working through this book,
you'll gain an
in-depth knowledge of the most important features of
Dreamweaver."
> 2. I am interested in purchasing this book mainly to
implement cool Ajax/Spry
> stuff on my e-commerce site. I am asking about PHP
because the book's content
> mentions making a PHP site. Now I realize this book is
not about shopping
> carts, but the Ajax/Spry stuff can be implemented on
e-commerce sites, right?
The book provides detailed coverage of the Spry 1.4 features
in
Dreamweaver CS3, which can be applied to any site. With minor
adaptation, you can also use Spry 1.6. It's important to
realize,
though, that the Spry/Ajax features in Dreamweaver CS3 do
*not* cover
asynchronous requests to the server. The ability to
communicate
asynchronously with the server was added in Spry 1.6. If it's
asynchronous server calls and responses that you're after,
your
programmer will need to dig around in the Spry 1.6
documentation on the
Adobe Labs site, and hand-code it.
The Spry features covered in the book are Spry Effects, Spry
UI widgets
(menu, accordion, tabbed panels, and collapsible panels),
Spry
Validation Widgets, and Spry XML Data Sets.
> I just want to know if I can learn and implement
Ajax/Spry apps on my
> simple HTML/CSS-based pages, on a server which allows
server-side PHP scripts
> but the pages themselves are not with .php extension. Is
this book still going
> to be applicable?
Yes. All the Spry techniques can be used on any web page. It
doesn't
need to be a PHP one.
David Powers, Adobe Community Expert
Author, "The Essential Guide to Dreamweaver CS3" (friends of
ED)
Author, "PHP Solutions" (friends of ED)
http://foundationphp.com/

Working with large data and PHP

Similar Messages

Maybe you are looking for