Very large HashMap/Hashtable

I have a situation where I cannot use a database or a file for field updation as it would cause serious problems related to the performance of the application. The application is run for long hours and when it is terminated some counters are written into a file. I am using some hashmaps. The problem is that the size of the hashmaps run into very big numbers (in the order of 10^10) and each cell of hashmap has a key which corresponds to a separate counter. Can I handle this large hashmap? Another problem is that it takes a long long time to write in the file when the application terminates. I noted that the Iterator runs into an interminable loop. Should I use different data structure?

A simple HashMap can only handle about 10^9 objects, however you can merge a number of HashMaps or use ConcurrentHashMap which does this to handle more objects.
If your object is about 200 bytes you will need about 3 TB (3,000 GB) of memory to hold these objects.
However, there are caching solutions which allow you to keep some of the objects in memory and some on disk, I.e. you could have about 1 GB of memory and about 3 TB of disk. The more memory you have the better. I am not sure which cache would be the best solution for this. I would search for reviews of java cache solutions.
Another option is that you store the data in a database and cache as many values as you can. Like Hibernate does.

Similar Messages

I received the error (in iCal on my iMac): "The server responded with an error". The error message is very large, and if there is a way to acknowledge and close it I can't find it. Because this error message is open, I can't do anything in iCal. Any sugge

I received the error (in iCal on my iMac): "The server responded with an error". The error message is very large, and if there is a way to acknowledge and close it I can't find it. Because this error message is open, I can't do anything in iCal. Any suggestions on how I could kill this error message? Thanks.
iMac, Mac OS X (10.7.2)
Basically i tried to enter too much information into my calendar and it has crashed now i can not get rid of the error message or use the calendar can anyone help please

did you find ou how to get rid of it i can't

What is the optimum core configuration for a new Mac Pro to process and manipulate very large (80 megapixel) images using PhotoShop and Camera Raw?

Hello:
I will be using creative techniques to process and manipulate a large number (hundreds) of very large (80 megapixel) images captured using a medium format digital back (Phase One IQ180).
Final output will be digital fine art imagery printed using an Epson 11880 at large sizes (up to 60 inches x ?), retaining the highest possible quality and resolution. I will be using Adobe CC PhotoShop and Camera RAW as well as Capture One software. PhotoShop filters will be used extensively.
The Mac Pro needs to be optimized for the above purpose and be useful for at least five years. I plan to max out all the other options (RAM, graphics cards, storage). Performance is more important than cost.
The few discussions I have found that mention optimum core configurations seem to lean toward 6 or 8 (but likely are not taking into consideration my need for manipulating a large number of very large files), so I am looking to this foum for opinions.
Thank you,
Kent

See if this helps
http://macperformanceguide.com/index_topics.html#MacPro2013

I was recently sent a very large emoji message amd now when i try to acces messages it will not display mesagges and then will bring me back to the home page

was recently sent a very large emoji message amd now when i try to acces messages it will not display mesagges and then will bring me back to the home page

Quit the messages app and reset your phone.
Go to the Home screen and double click the Home button. That will reveal the row of recently used apps at the bottom of the screen. Tap and hold on the app in question until it jiggles and displays a minus sign. Tap the minus sign to actually quit the app. Then tap anywhere on the screen above that bottom row to return the screen to normal. Then restart the app and see if it works normally. Then reset your device. Press and hold the Home and Sleep buttons simultaneously ignoring the red slider should one appear until the Apple logo appears. Let go of the buttons and let the device restart.
After the phone restarts go into Messages and delete the thread containing the problem message.

In Mail, one mailbox for Recovered Message (AOL) keeps showing 1 very large message that I cannot delete. How can I get rid of this recurring problem, please?

In Mail on iMac, successfully running OS X Lion, one mailbox on My Mac for "Recovered Messages (from AOL)" keeps showing 1 very large message (more than 20 Mb) that I just cannot seem to delete. Each time I go into my In Box, the "loading" symbol spins and the message appears in the "Recovered Messages" mailbox. How can I get rid of this recurrent file, please?
At the same time, I'm not receviving any new mails in my In Box, although, if I look at the same account on my MacBook Pro, I can indeed see the incoming mails (but on that machine I do not have the "recovery" problem).
The help of a clear-thinking Apple fan would be greatly appreciated.
Many thanks.
From Ian in Paris, France

Ian
I worked it out.
Unhide your hidden files ( I used a widget from http://www.apple.com/downloads/dashboard/developer/hiddenfiles.html)
Go to your HD.
Go to Users.
Go to your House (home)
there should be a hidden Library folder there (it will be transparent)
Go to Mail in this folder
The next folder ( for me ) is V2
Click on that and the next one will be a whole list of your mail servers, and one folder called Mailboxes
Click on that and there should be a folder called recovered messages (server) . mbox
Click on that there a random numbered/lettered folder -> data
In that data folder is a list of random numbered folders (i.e a folder called 2, one called 9 etc) and in EACH of these, another numbered folder, and then a folder called messages.
In the messages folder delete all of the ebmx (I think that's what they were from memory, sorry I forgot as I already deleted my trash after my golden moment).
This was GOLDEN for me. Reason being, when I went to delete my "recovered file" in mail, it would give me an error message " cannot delete 2500 files". I knew it was only 1 file so this was weird. Why 2500 files? Because if you click on the ebmx files like I did, hey presto, it turned out that they were ALL THE SAME MESSAGE = 2500 times. In each of those folders in the random numbers, in their related message folder.
Now remember - DONT delete the folder, make sure you have gone to the message folder, found all those pesky ebmx files and deleted THOSE, not the folder.
It worked for me. No restarting or anything. And recovered file. GONE.
Started receiving and syncing mail again. Woohoo.
Best wishes.

How do I stop itunes from downloading a very large apps that I downloaded direct to my iphone?

To get around my 5GB limit for internet service, I download a very large (350MB) dictionary app directly to my iphone using a wireless website run by a merchant in town. Now every time I run itunes, it wants to download the app. Is there any way I can stop itunes from attempting the download?

Some types of issues, such as that one, can't be resolved by the volunteer support here. If I think a request can be answered in the boards but was posted in the wrong part of them, I'll just ignore the fact it's in the wrong place.
(65827)

Is there a way to split a very large 1 page pdf into letter size multiple page pdf?

I often have very large single page pdfs that need to be printed onto letter size paper. Usually I don't have access to the printer where I'm working so I have to send the file to someone for printing.
I have AXI pro, they don't.
I want to make sure the job is printed as I specify and most of the users are using Reader. So I want to give the someone the pdf ready to print sized in legal. This requires manipulation of the pdf that I don't seem to be able to figure out how to do.
In older versions of Acrobat, I could print to a new pdf and designate the page size. Acrobat would create the multipage pdf. The newer versions don't allow this.
With OSX 10.8 & AXI you can't save, export, split a one page (68" x 16") document into multiple page letter size (16 pages) pdf.
Perhaps this can be done by printing to eps and running through distiller again or something else, but I'm stumped at the moment.
Any suggestions of how to attach this would be appreciated.
Thanks.

That's a tough one. Acrobat is not designed for tiling PDF files to create another PDF. That's really what you're asking. There is the option to PRINT to a PDF, and turn on the Poster feature. If were in Windows where there is a real Adobe PDF printer driver, you could probably use that feature. But for various reasons (too complicated to describe here), that was withdrawn on the Macintosh.
If you have a copy of Adobe InDesign, and if you installed an Adobe PDF 9 PPD file (see description below), it could be done in a somewhat awkward way. InDesign allows you to place PDF files so you would need to make a page of the proper size and place your large PDF:
Then after installing the Adobe PDF 9 PPD file, you could choose File > Print. Then choose to print a PostScript file to the Adobe PDF 9.0 PPD file. In the Setup panel, you'd choose a Letter size page. Then you'd choose the Tile option at the bottom and set the Overlap amount:
Then you'd save the PostScript file and process through Distiller.
My blog post below describes how to find and install the Adobe PDF 9.0 PPD file:
http://indesignsecrets.com/creating-postscript-files-in-snow-leopard-for-older-print-workf lows.php

Profile Performance and Memory shows very large 'VI Time' value

When I run the Profile Performance and Memory tool on my project, I get very large numbers for VI Time (and Sub VIs Time and Total Time) for some VIs. For example 1844674407370752.5. I have selected only 'Timing statistics' and 'Timing details'. Sometimes the numbers start with reasonable values, then when updating the display with the snapshot button they might get large and stay large. Other VI Times remain reasonable.
LabVIEW 2011 Version 11.0 (32-bit). Windows 7.
What gives?
- les

les,
the number indicates some kind of overroll.... so, do you have a vi where this happens all the time? Can you share this with us?
thanks,
Norbert
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.

Slow Performance or XDP File size very large

There have been a few reports of people having slow performance in their forms (tyically for Dynamic forms) or file sizes of XDP files being very large.
These are the symptoms of a problem with cut and paste in Designer where a Process Instruction (PI) used to control how Designer displays a specific palette is repeated many many times. If you look in your XDP source and see this line repeated more than once then you have the issue:
The problem has been resolved by applying a style sheet to the XDP and removing the instruction (until now). A patch has been released that will fix the cut and paste issue as well as repair your templates when you open them in a designer with the patch applied.
Here is a blog entry that describes the patch as well as where to get it.
http://blogs.adobe.com/livecycle/2009/03/post.html

My XDP file grow up to 145mb before i decided to see what was actually happening.
It appears that the LvieCycle Designer ES program sometimes writes alot of redundant data... the same line millions of times over & over again.
I wrote this small java program which reduced the size up to 111KB !!!!!!!!!!!!!!!!!! (wow what a bug that must have been!!!)
Here's the sourcecode:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class MakeSmaller {
private static final String DELETE_STRING = "                           <?templateDesigner StyleID aped3?>";
public static void main(String... args) {
BufferedReader br = null;
BufferedWriter bw = null;
try {
   br = new BufferedReader(new FileReader(args[0]));
   bw = new BufferedWriter(new BufferedWriter(new FileWriter(args[0] + ".small")));
   String line = null;
   boolean firstOccurence = true;
   while((line = br.readLine()) != null) {
    if (line.equals(DELETE_STRING)) {
     if (firstOccurence) {
      bw.write(line + "\n");
      firstOccurence = false;
    } else {
     bw.write(line + "\n");
     firstOccurence = true;
} catch (FileNotFoundException e) {
   e.printStackTrace();
} catch (IOException e) {
   e.printStackTrace();
} finally {
   if (br != null) {
    try {
     br.close();
    } catch (IOException e) {
     e.printStackTrace();
   if (bw != null) {
    try {
     bw.close();
    } catch (IOException e) {
     e.printStackTrace();
File that gets generated is the same as the xdp file (same location) but gets the extension .small. Just in case something goes wrong the original file is NOT modified as you can see in the source code. And yes Designer REALLY wrote that line like a gazillion times in the .xdp file (shame on the programmers!!)
You can also see that i also write the first occurrence to the small file just in case its needed...

When writing an email the letters are so small that I can hardly read them. If I increase the size then the outgoing email letters are very large.

letters are very large. I have tried changing the font but it doesn't help. This is very annoying. How can I remedy this?

I use these settings, so try them:
Tools > Options > Display > Formatting tab
or
Menu icon > Options > Options > Display > Formatting tab
* eg: Default font: Arial
* size 14
Note: this sets the default font to be used in both display of messages and the font to use as default in composing messages.
Under Plain text Messages
* select: Style : Regular Size: Regular and colour Black
* Click on 'Advanced' button
* make sure all sizes - 14
* select; 'allow messages to use other fonts'
* Select; 'Use fixed width fonts for plain text messages'
* click on OK
Note: This will use the default eg: Arial to be used, but allows the display of messages to use other fonts if the sender specified them.
then at top click on 'Composition' > General tab
note: This sets the default composing text colour and background colour.
under HTML
* Font: Variable width
* Size : Medium
* Select a text colour for writing emails eg: Blue
* Select background colur eg: white
click on OK to save and close Options.

How can we suggest a new DBA OCE certification for very large databases?

How can we suggest a new DBA OCE certification for very large databases?
What web site, or what phone number can we call to suggest creating a VLDB OCE certification.
The largest databases that I have ever worked with barely over 1 Trillion Bytes.
Some people told me that the results of being a DBA totally change when you have a VERY LARGE DATABASE.
I could guess that maybe some of the following topics of how to configure might be on it,
* Partitioning
* parallel
* bigger block size - DSS vs OLTP
* etc
Where could I send in a recommendation?
Thanks Roger

I wish there were some details about the OCE data warehousing.
Look at the topics for 1Z0-515. Assume that the 'lightweight' topics will go (like Best Practices) and that there will be more technical topics added.
Oracle Database 11g Data Warehousing Essentials | Oracle Certification Exam
Overview of Data Warehousing
Describe the benefits of a data warehouse
Describe the technical characteristics of a data warehouse
Describe the Oracle Database structures used primarily by a data warehouse
Explain the use of materialized views
Implement Database Resource Manager to control resource usage
Identify and explain the benefits provided by standard Oracle Database 11g enhancements for a data warehouse
Parallelism
Explain how the Oracle optimizer determines the degree of parallelism
Configure parallelism
Explain how parallelism and partitioning work together
Partitioning
Describe types of partitioning
Describe the benefits of partitioning
Implement partition-wise joins
Result Cache
Describe how the SQL Result Cache operates
Identify the scenarios which benefit the most from Result Set Caching
OLAP
Explain how Oracle OLAP delivers high performance
Describe how applications can access data stored in Oracle OLAP cubes
Advanced Compression
Explain the benefits provided by Advanced Compression
Explain how Advanced Compression operates
Describe how Advanced Compression interacts with other Oracle options and utilities
Data integration
Explain Oracle's overall approach to data integration
Describe the benefits provided by ODI
Differentiate the components of ODI
Create integration data flows with ODI
Ensure data quality with OWB
Explain the concept and use of real-time data integration
Describe the architecture of Oracle's data integration solutions
Data mining and analysis
Describe the components of Oracle's Data Mining option
Describe the analytical functions provided by Oracle Data Mining
Identify use cases that can benefit from Oracle Data Mining
Identify which Oracle products use Oracle Data Mining
Sizing
Properly size all resources to be used in a data warehouse configuration
Exadata
Describe the architecture of the Sun Oracle Database Machine
Describe configuration options for an Exadata Storage Server
Explain the advantages provided by the Exadata Storage Server
Best practices for performance
Employ best practices to load incremental data into a data warehouse
Employ best practices for using Oracle features to implement high performance data warehouses

Upload very large file to SharePoint Online

Hi,
I tried uploading very large files to SharePoint online via the REST API using JAVA. Uploading files with a size of ~160 mb works well, but if the filesize is between 400mb and 1,6 gb I either receive a Socket Exception (Connection reset by peer) or an error
of SharePoint telling me "the security validation for the page is invalid".
So first of all I want to ask you, how to work around the security validation error. As far as I understand the token, which I have added to my X-RequestDigest header of the http post request seems to be expired (by default it expires after 1800 seconds).
Uploading such large files is time consuming, so I can't change the token in my header while uploading the file. How can I extend the expiration time of the token (if it is possible)? Could it help to send post requests with the same token
continuously while uploading the file and therefore prevent the token from expiration? Is there any other strategy to upload such large files which need much more than 1800 seconds for upload?
Additionally any thoughts on the socket exception? It happens quite sporadically. So sometimes it happens after uploading 100 mb, and the other time I have already uploaded 1 gb.
Thanks in advance!

Hi,
Thanks for the reply, the reason I'm looking into this is so users can migrate there files to SharePoint online/Office 365. The max file limit is 2gb, so thought I would try and cover this.
I've looked into that link before, and when I try and use those end points ie StartUpload I get the following response
{"error":{"code":"-1, System.NotImplementedException","message":{"lang":"en-US","value":"The
method or operation is not implemented."}}}
I can only presume is that these endpoints have not been implemented yet, even though the documentation says they only available for Office 365 which is what I am using.
Also, the other strange thing is that I can actually upload a file of 512mb to the server as I can see it in the web UI. The problem I am experiencing is get a response, it hangs on this line until the TimeOut is reached.
WebResponse wresp = wreq.GetResponse();
I've tried flushing and closing the stream, sendchunked but still no success

Operations on very large binary numbers

Hi guys,
I'm trying to write a java class for manipulations of very large binary numbers.
I'm representing the BN internally as boolean[] (array of booleans false for 0 & true for 1).
I want to write an algorithm for the following operations :
shifLeft (boolean[] b,int n) : shifting binary number to left by n positions.preferably a circular shift .
and also the corresponding shiftRight() method.
can some one guide me on how to implement this ?
thanks.

Consider an array of ascii chars....
array => | |a|b|c|d|
offset = 1
len = 4In the above there are actually five spots in the
array but the offset points to the second position in
the array and the length is 4.
If I was to extract the value it would be "abcd"
because of the offset and the length.
Now a 'shift right' means that if it is "abcd" then
it should now be "abc" (because the d fell off the
end.)
I can do that like this.
array => | |a|b|c|d|
offset = 1
len = 3Notice in the above that nothing changed except the
length. But because the length changed if I
extracted the value I would get "abc" because the
offset is 1 and the length is three.
Notice also that there was no array copying.Are you sure shifting works like that ?
what I know about left shifting is that the 'd' should be moved left -toghether with a, b, c- and we append a 0 in place of 'd' .
from what you said the 'd' would be gone , or am I again wrong ?
it seems i still didn't get any satisfactory answer to this problem...

Setting resolution, deciding file type, for very LARGE Canvas prints. 36MP camera.

Okay, so I noticed my lightroom was on 240 PPI resolution. I changed it to 300 because I read 300 was for standard prints. What would I need to set the export to in the resolution box for a very large Canvas?
Is it better to choose Tiff instead of Jpeg for prints of this quality, if not then what should I choose?
I am using a Sony A7R full frame 36.4MP and with some of the sharp Zeiss lens there is really no pixelation that noticeable when I zoom in 100 percent. Of course the A7R is being said to be one of the best Sensors on the market today. It's supposed to be like the Nikon D800E, but apparently it has some advantages.
In other words, I want to export in as high of quality as possible for the Canvas. File size is not an issue.

Changing the resolution setting does absolutely nothing to the digital image. This is a common misconception. The only thing that counts in the digital image is the pixel dimensions. Regardless of what the PPI setting is (240, 300, 600, whatever) the image still has the same number of pixels. To determine what you need for a print of any size it is necessary to multiply the inches by the desired pixels per inch. Suppose you want a 16 x 20" print at 300 pixels per inch. The math would be something like this:
300x16 = 4800 pixels
300x20 = 6000 pixels
So to print a 16 x 20" print you would need an image that is 4800 x 6000 pixels. And the PPI setting can be anything you want it to be because it has no effect on the image.

I need to sort very large Excel files and perform other operations. How much faster would this be on a MacPro rather than my MacBook Pro i7, 2.6, 15R?

I am a scientist and run my own business. Money is tight. I have some very large Excel files (~200MB) that I need to sort and perform logic operations on. I currently use a MacBookPro (i7 core, 2.6GHz, 16GB 1600 MHz DDR3) and I am thinking about buying a multicore MacPro. Some of the operations take half an hour to perform. How much faster should I expect these operations to happen on a new MacPro? Is there a significant speed advantage in the 6 core vs 4 core? Practically speaking, what are the features I should look at and what is the speed bump I should expect if I go to 32GB or 64GB? Related to this I am using a 32 bit version of Excel. Is there a 64 bit spreadsheet that I can us on a Mac that has no limit on column and row size?

Grant Bennet-Alder,
It’s funny you mentioned using Activity Monitor. I use it all the time to watch when a computation cycle is finished so I can avoid a crash. I keep it up in the corner of my screen while I respond to email or work on a grant. Typically the %CPU will hang at ~100% (sometimes even saying the application is not responding in red) but will almost always complete the cycle if I let it go for 30 minutes or so. As long as I leave Excel alone while it is working it will not crash. I had not thought of using the Activity Monitor as you suggested. Also I did not realize using a 32 bit application limited me to 4GB of memory for each application. That is clearly a problem for this kind of work. Is there any work around for this? It seems like a 64-bit spreadsheet would help. I would love to use the new 64 bit Numbers but the current version limits the number of rows and columns. I tried it out on my MacBook Pro but my files don’t fit.
The hatter,
This may be the solution for me. I’m OK with assembling the unit you described (I’ve even etched my own boards) but feel very bad about needing to step away from Apple products. When I started computing this was the sort of thing computers were designed to do. Is there any native 64-bit spreadsheet that allows unlimited rows/columns, which will run on an Apple? Excel is only 64-bit on their machines.
Many thanks to both of you for your quick and on point answers!

Very large HashMap/Hashtable

Similar Messages

Maybe you are looking for