How to load file with 500M into noncontiguous memory and visit them

My data file consist of  a serial of 2d images. Each image is 174*130*2 butes and the total images in the file is up to 11000, which takes almost 500M bytes. I hope to load them into memory in one turn since I need to visit either image many times in random order during the data processing. When I load them into one big 3d array, the error of "out of memory" happended frequenctly.
First I tried to use QUEUE to load big chunk of data into noncontiguous memory. Queue structure is a good way to load and unload data into noncontiguous memory. But it dosn't fit here since I may need to visit any of the all the images at anytime. And it is not pratical if I dequeue one image and enqueue it into the opposite end, since the image visited may be not in sequential order.
Another choice is to put the whole file into multiple small arrays. In my case, I may load the data file into 11000 small 2d arrays and each array holds the data of one image. But I don't know how to visit these 2d array and I didn't get any cues in the posters here.
Any suggestion?
PC with 4G physical memory, Labview 7.1 and Win XP.
Thanks.

I'll try to get the ball rolling here -- hopefully there's some better ideas than mine out there.
1. I've never used IMAQ, but I suspect that it offers more efficient methods than stuff I dream up in standard LabVIEW.  Of course, maybe you don't use it either -- just sayin'...
2. Since you can't make a contiguous 11000-image 3D array, and don't know how to manage 11000 individual 2D arrays, how about a compromise?  Like, say, 11 3D arrays holding 1000 images each?   Then you just need ~50 MB chunks of contiguous memory.
3.  I'm picturing a partially hardcoded solution in which there are a bunch of cloned "functional globals" which each hold 1000 images of data.  This kind of thing can be done using (A) Reentrant vi's or (B) Template vi's.  You may need to use VI Server to instantiate them at run-time.
4. Then I'd build an "Action Engine" wrapper around that whole bunch of stuff.  It'd have an input for image #, for example, image # 6789.  From there it would do a quotient & remainder with 1000 to split 6789 into a quotient of 6 and remainder of 789.  Then the wrapper would access function global #6 and address image # 789 from it.
5. Specific details and trade offs depend a lot on the nature of your data processing.  My assumption was that you'd primarily need random access to images 1 or 2 at a time. 
I hope to learn some better ideas from other posters...
-Kevin P.
P.S.  Maybe I can be one of those "other posters" -- I just had another idea that should be much simpler.  Create a typedef custom control out of a cluster which contains a 2D image array.  Now make an 11000 element array of these clusters.  If I recall correctly, LabVIEW will allocate 11000 pointers.  The clusters that they point to will *not* need to be contiguous one after the other.  Now you can retrieve an image by indexing into the array of 11000 clusters and simply unbundling the 2D image data from the cluster element.

Similar Messages

  • How to attach file with an email in travel and expenses

    Hi Experts,
    I am new to sap cloud application studio.
    We have requirement that need to attach any type of files in the mail,
    I have gone through this : How to send an email with attachments in SAP Cloud Application Studio
    But compilation error at import AP.FO.ACTIVITY.GLOBAL., so not able to work with that lines of code.
    I am using these lines of code to send email.
    EMailUtilities.SendEMail(lang, bcc, cc, toAddress, htmlContent, from name, subject);
    So am not able to getting how to attach file with this code.
    Please help me regarding this.
    Thanks & Regards,
    Preethi Ande

    Hi Preethi,
         I am trying to send mail from custom BO in Travel on Demand System.I am getting  compilation error.please look into the attached screenshot for the code.
    Please assist me on this,                                Thanks in advance
    regards,
    Sendhilazhagan

  • How to load file with seperators

    Hi, I am attempting to load a CSV file. Currently the first column has data in the format '101-000-12345-345' where
    101 is the entity
    000 is the Custom1
    12345 is account
    345 is ICP. There are other columns in the file for C2, C3 etc. My problem is that I need help on how to define an Import format which can (in the above example) pick up 101 for entity, 000 for Custom 1 etc. Please note the length for ENtity, C1, Account and ICP is fixed.
    Please advise

    Hi,
    It sounds like you need to define "-" as your delimiter for that specific Import Group.
    When you first create your Import Group you will need to define the delimiter. Once you have done this, you can then begin to define your Dimensions - here you are defining the format of your datafile. You have a couple options here - my recommendation would be to use the Import Format Builder - it will let you drag and drop sections to define your dimension. This will get you a bit more familiar with how this works.
    I would strongly recommend taking a look at the FDM Admin Guide, there is an entire section dedicated to creating an Import Group. Here is the link to the 11.1.2.2 doc - though you may be on a different version: http://docs.oracle.com/cd/E17236_01/epm.1112/fdm_admin.pdf
    Good luck!

  • How do I split each page into a PDF and name them after a line of text on the split page?

    All,
    I am running a report that is creating hundreds of letters to be sent to customers.  When I run the report it creates one large PDF (one letter = one page).  To help with a specific business solution I need to split each page into an individual PDF.  I know I can extract the pages using the extraction tool.  I have also been able to duplicate this from the Javascript console, but only with generic names.  The problem is that I need each PDF named after the customer's account number.  The account number is listed on the page that I am extracting, but I am having a hard time parsining the text on the pages to find it.
    I need to be able to turn each page into a seperate PDF and then rename that PDF to match the account number list on the page.
    The account number, is of course, different for every customer and has varied prefixes.  The line of text that I need to read starts like this:
    Account Number#: NJA00001
    I need to just pull the NJA00001.
    I would really love to automate this process.
    Can you help me with this?  Thanks in advance.
    PS: I am a little rusty on my Javascript, its been a couple years.

    This can probably be achieved with a custom-made script, but it's not possible to know for sure before seeing some sample files.

  • How to load CSV with accented characters?

    Hi,
    I have a database instance with NLS_CHARACTERSET = 'AL32UTF'. I upload a csv file with APEX into WW_FLOW_FILES table and then parse it with dbms_lob.instr. The problem that I am facing is that if I save the CSV file in UTF8, then this works fine, if I save it in any other encoding then it's displaying 'funny' characters when I try to parse it and display the parsing result and the source data contains some accented characters, like, á, é, ő etc,
    I am not sure if this is an APEX issue or I rather be turning to NLS forum. TIA.
    Tamas
    Edited by: Tamas Szecsy on Jan 30, 2011 6:34 PM

    Did you every find a resolution to this issue on importing with accented characters?

  • Sum Encrypt does not recognize a copy of a file through linux. How decrypt this file with Sum Encrypt?

    Sum Encrypt does not recognize a copy of a file through linux to Mac 0S9. How decrypt this file with Sum Encrypt?

    Hello, and welcome to Apple Support Communities!
    I am not familiar with the program that you list, however files in OS 9 use things called Resource Forks (rather than file extensions) for filetype and creator.
    If you sent a file through Linux these resource forks probably got stripped and now the file is nothing to your Macintosh.
    It is best to BinHex your files (.hqx) before leaving the HFS file system so that their resource forks are preserved.
    Regards,
    Ryan

  • How to share files with iOS and android phone with my iPhone 6

    I Just wanna know how to transfer files from iPhone to android phones and vise versa.

    Best bet would be to use a third party cloud service such as DropBox or Box. Apps for them should be available for both platforms (they definitely are for iOS).

  • URGENT: Problems Loading files with SQL Loader into a BLOB column

    Hi friends,
    I read a lot about how to load files into blob columns, but I found errors that I can't solve.
    I've read several notes in these forums, ine of them:
    sql loader: loading external file into blob
    and tried the solutions but without good results.
    Here are some of my tests:
    With this .ctl:
    LOAD DATA
    INFILE *
    INTO TABLE mytable
    REPLACE
    FIELDS TERMINATED BY ','
    number1 INTEGER EXTERNAL,
    cad1 CHAR(250),
    image1 LOBFILE(cad1) TERMINATED BY EOF
    BEGINDATA
    1153,/opt/oracle/appl/myapp/1.0.0/img/1153.JPG,
    the error when I execute sqlldr is:
    SQL*Loader-350: Syntax error at line 9.
    Expecting "," or ")", found "LOBFILE".
    image1 LOBFILE(cad1) TERMINATED BY EOF
    ^
    What problem exists with LOBFILE ??
    (mytable of course has number1 as a NUMBER, cad1 as VARCHAR2(250) and image1 as BLOB
    I tried too with :
    LOAD DATA
    INFILE sample.dat
    INTO TABLE mytable
    FIELDS TERMINATED BY ','
    (cad1 CHAR(3),
    cad2 FILLER CHAR(30),
    image1 BFILE(CONSTANT "/opt/oracle/appl/myapp/1.0.0/img/", cad2))
    sample.dat is:
    1153,1153.JPEG,
    and error is:
    SQL*Loader-350: Syntax error at line 6.
    Expecting "," or ")", found "FILLER".
    cad2 FILLER CHAR(30),
    ^
    I tried too with a procedure, but without results...
    Any idea about this error messages?
    Thanks a lot.
    Jose L.

    > So you think that if one person put an "urgent" in the subject is screwing the problems of
    other people?
    Absolutely. As you are telling them "My posting is more important than yours and deserve faster attention and resolution than yours!".
    So what could a typical response be? Someone telling you that his posting is more important by using the phrase "VERY URGENT!". And the next poster may decide that, no, his problem is evern more import - and use "EXTREMELY URGENT!!" as the subject. And the next one then raises the stakes by claiming his problem is "CODE RED! CRITICAL. DEFCON 4. URGENT!!!!".
    Stupid, isn't it? As stupid as your instance that there is nothing wrong with your pitiful clamoring for attention to your problem by saying it is urgent.
    What does the RFC's say about a meaningful title/subject in a public forum? I trust that you know what a RFC is? After all, you claim to have used public forums on the Internet for some years now..
    The RFC on "public forums" is called The Usenet Article Format. This is what it has to say about the SUBJECT of a public posting:
    =
    The "Subject" line (formerly "Title") tells what the message is about. It should be suggestive enough of the contents of the message to enable a reader to make a decision whether to read the message based on the subject alone. If the message is submitted in response to another message (e.g., is a follow-up) the default subject should begin with the four characters "Re: ", and the "References" line is required. For follow-ups, the use of the "Summary" line is encouraged.
    =
    ([url http://www.cs.tut.fi/~jkorpela/rfc/1036.html]RFC 1036, the Usenet article format)
    Or how about [url http://www.cs.tut.fi/~jkorpela/usenet/dont.html]The seven don'ts of Usenet?
    Point 7 of the Don'ts:
    Don't try to catch attention by typing something foolish like "PLEASE HELP ME!!!! URGENT!!! I NEED YOUR HELP!!!" into the Subject line. Instead, type something informative (using normal mixed case!) that describes the subject matter.
    Please tell me that you are not too thick to understand the basic principles of netiquette, or to argue with the RFCs that governs the very fabric of the Internet.
    As for when I have an "urgent" problem? In my "real" work? I take it up with Oracle Support on Metalink by filing an iTAR/SR. As any non-idiot should do with a real-life Oracle crisis problem.
    I do not barge into a public forum like you do, jump up and down, and demand quick attention by claiming that my problem is more important and more urgent and more deserving of attention that other people's problem in the very same forum.

  • How to load budgeted sales dollars into SAP.

    hello experts
    we are using ecc 5.0
    pl advise how how to load budgeted sales dollars into SAP
    thnks
    vinod

    Hello,
    Budgets are like threshold figures where your actuals cannot exceed your budgets.
    Whereas, planned data is for variance analysis with your actuals.
    You can configure the cost center planning so that the user can upload the data through KP06 or KP26
    This is controlled by planning are in your planning profile. You need to use "Cost ctrs: Cost element/activity inputs" planning area for this purpose.
    Hope you are conversant with Integrated excel upload.
    To do integrated excel upload of cost center plan data.
    IMG ==> Controlling ==> Cost Center Accounting ==> Planning ==> Manual Planning User Defined Layouts - Create your own layout or copy from standard layout and change it.
    IMG ==> Controlling ==> Cost Center Accounting ==> Planning ==> Define User Defined Planner Profiles
    Copy the SAPALL to ZSAPALL. You cannot change SAPALL as it is standard.
    In ZSAPALL planner profile, select proper planning area and enter your layout.
    Tick Integrate Excel.
    Once the popup for KP06 comes, you need enter the data and save it. It would generate the file name.
    With the same file name you need upload the plan data by using KP06.
    In KP06
    Extras ==> Excel Planning ==> Upload
    Select the file as prepared in accordance with format at the time of generating the file name.
    Hope this solves your problem.
    Please let me know if you need futher help.
    Regards,
    Ravi

  • HT1386 I have sync an ipad with a new computer and the photos have been transfered into files with no extensions. I can view them on the ipad but canot view and delete from the computer.

    I have synced a new computer with the ipad and the photos have been transferred as a file with no exstention. I cannot view them on the computer but can on the ipad.

    Well I forget the exact keys, but believe pressing and holding down CTRL before clicking Restore will allow you to browse and find the backup file (may be wrong).
    In the end, as far as the iPhone thinks, it is a new computer so it will erase and resync stuff and as long as you loaded everything back into iTunes all should sync back. Main issue is other stuff like the data within the apps, notes, etc that are part of the backup which is why you would want to try to restore from the backup. If you don't mind losing data in apps (eg: do you care you have to relog into some apps and start over, don't care about game scores, etc) then you don't have to worry about the backup. If you have important data in some of those apps then you will need to restore from the backup.

  • How to load data from Quikbooks into PeopleSoft General Ledger module ?

    Folks,
    Hello.
    My client's data is currently stored in Quikbooks.
    After install PeopleSoft FSCM 9.0, I need to load the data from Quickbooks into PeopleSoft General Ledger module. But I don't know how to use PeopleSoft Integration Broker to integrate with Quikbooks.
    Can any folks tell me how to load data from Quikbooks into PeopleSoft General Ledger module ?

    Hi,
    If the data load is one time process, then you can use PL/SQL or datamover to load the data to PeopleSoft application.
    If you want to load the data in real time, then you need to create In-bound and out-bound nodes to perform the transaction.
    Thanks
    Soundappan

  • Load File with wildcards?

    Hello,
    how could I load files with this pattern:
    ausw_[name]_[date].xls (without the brackets)
    Workbook workbook = Workbook.getWorkbook(new File("C:\\extracts/ausw_[name]_[date].xls"));
    Could it be work with Regular Expressions?
    Has anyone an idee?
    Greetings
    Marco

    Check java.io.FilenameFilter and java.io.File.listFiles (FilenameFilter) or search the forum.

  • How to load Essbase member formula into DRM using Automator

    Hi,
    I just wonder how to load Essbase member formula into a member property in DRM?
    I encounter a problem when loading a member formula which as new line in the formula.
    Here is the example of the record for Automator using | (pipe) delimiter where the formula has carriage return / new line:
    ChangeProp|Master|Accounts|CYTDSignOnBonus|ESB_Formula|[OpenInputValueBlock]
    [CYTD("Sign On Bonus")]
    [CloseInputValueBlock]
    In the above example. when the formula is loaded, it only load the 1st line and strip the remaining line.
    Thanks.

    Hello,
    I have encountered this issue as well and here is how I receommend you deal with it. Unfortunately, DRM does not understand multiple lines in a formula and so it truncates the other lines.
    1. The best way around it is to ensure that your formula does not have multiple lines. Strip all the enters and only use spaces. Essbase does not care about aesthetics as long as the formula is all in one line. This should take care of it.
    2. Sometimes, the formulas for member properties are very long and they are in multiple lines. You can try using excel formulas to bring them all in one line. After removing enters and tabs myself multiple times, I informed my Essbase developers of this DRM issue and told them to provide me with the formulas without using Enters and tabs.
    Hope this helps.
    -- Adi

  • How to load Matrix report data into basic table data using ODI

    Hi,
    How to load Matrix report data into basic table data using oracle Data Integrator ?
    Requirement Description:
    Following is the matrix report data:
    JOB                       DEPT10                DEPT20 
    ANALYST                                           6000
    CLERK                   1300                     1900 Need to convert it into below format:
    JOB                             Dept                        Salary
    ANALYST                  DEPT10     
    ANALYST                  DEPT20                     6000
    CLERK                       DEPT10                    1300
    CLERK                       DEPT20                    1900
        Thanks for your help in advance. Let me know if any further explanation is required.

    Your list seems to be a little restrictive, you can do a lot more with ODI procedures.
    If you create new procedure, and add a step. In the 'command on source' tab set you technology and schema as per your source database. Use the unpivot functionality as described in the link, please, rather than using 'SELECT *' use the appropriate column names and alias them for eg:
    SELECT job as job,
    deptsal as deptsal,
    saldesc as saledesc
    FROM pivoted_data
    UNPIVOT (
    deptsal --<-- unpivot_clause
    FOR saldesc --<-- unpivot_for_clause
    IN (d10_sal, d20_sal, d30_sal, d40_sal) --<-- unpivot_in_clause
    Then in your 'command on target' tab set the technology and schema to your target db, then put your INSERT statement for eg:
    INSERT INTO job_sales
    (job,
    deptsal,
    saledesc
    VALUES
    :job,
    :deptsal,
    :saledesc
    Therefore you are using bind variables from source to load data into target.
    Obviously if the source and target table are in the same database, then you can have it all in one statement in the 'command on target' as
    INSERT INTO job_sales
    (job,
    deptsal,
    saledesc
    SELECT job as job,
    deptsal as deptsal,
    saldesc as saledesc
    FROM pivoted_data
    UNPIVOT (
    deptsal --<-- unpivot_clause
    FOR saldesc --<-- unpivot_for_clause
    IN (d10_sal, d20_sal, d30_sal, d40_sal) --<-- unpivot_in_clause
    also set the log counter as 'Insert' on the tab where your INSERT statement is, so you know how many rows you insert into the table.
    Hope this helps.
    BUT remember that this feature only came out in Oracle 11g.

  • How to load the certificate authority into the keystore for the weblogic8.1

    how to load the certificate authority into the keystore for the weblogic8.1
    ==================================================
    Getting the message below when trying to improt the certificate to the weblogic 8.1 web server. Received this certificate from our internal IT certificate authority. Trying to import the certificate to our test sytem.
    ===================================================
    keytool error: java.lang.Exception: Failed to establish chain from reply
    Import failed. Verify that the Certificate Authority that signed 'certi.pem'
    has been loaded into your keystore 'keystore\pskey'
    To view keystore contents issue 'PSkeymanager -list -keystore keystore\pskey [-v
    To preview a certificate file issue 'PSkeymanager -previewfilecert -file certi.pem'

    You need to populate that field using cmod code. Find out from which table that field is and go to transaction cmod then enter project name and select component radio button then display.
    Now select the FM EXIT_SAPLRSAP_001  if your datasource is transactional dataource
    EXIT_SAPLRSAP_002 for master data attibute
    EXIT_SAPLRSAP_003 for Hierarchies
    EXIT_SAPLRSAP_004 for text
    then populate code .
    After your code then delete data from ods then reinit to populate the enhanced field.
    Hope it helps..

Maybe you are looking for

  • Threads and lockups

    Hi everyone, I have a question about threads. I have a JDesktopPane in a Frame. I have added a number of JInternalFrame to the JDeskTopPane that is show/hide as required. This all works fine. One particular internal frame is a frame that allows the t

  • When synching my IPhone 3Gs I now get a sign "Attempting to copy to Disk failed. The required disk cannot be found." This is after synching a new Iphone 4s.. Help!!

    When attempting to synch my IPhone 3Gs I now get a sign "Attempting to copy to Disk failed. The required disk cannot be found." This is after synching a new Iphone 4s and having received some help from Apple Support who assisted with some setting cha

  • Problem opening a web template!

    Hi Experts   When I open a Web Templates, it says that it has a HTML Fragments & cannot be edited in WAD as it is not designed in such a way to edit only in HTML editor??? Any ideas are highly appreaciated. Thanks BI Learner

  • Machine that gets Time Machine Error ALL THE TIME

    Lets try and list out all the machines that cannot get Time Machine to work all the time. Here is mine: Machine: 1st Gen Macbook Pro (June 06) with upgraded HDD (160GB) and 2GB RAM TM HDD Tried: ALL external HDD (USB and Firewire) Error: The back up

  • ICCMP_INBOX

    hi all, I have a requirement where by which i need to add a Z field to the InboxItems view of the ICCMP_INBOX component. i.e., when i go to the inbox of the interaction Center and search for a mail, the results that would be in the result list, shoul