An effective way to compress Large Object Heap

I think I've found an effective way to compress LOH:
If CLR always alloc every large object at the beginning of a RAM page(usually 4KB per page)，then the large object heap(LOH) can be compressed without much
cost: CLR can compress LOH by modifying RAM page table and TLB instead of copying data. If so, small fragmentation maybe still exist (less then a memory page size per fragment), but there would be no large fragmentation, and compressing would be very fast
because of no copying. To do this, OS support may be needed, fortunately Windows OS and Visual Studio are both Microsoft's softwares, so Microsoft can implement this at least on Windows.

I suspect that this technique is already used in a 3rd party implementation of Java. It was briefly mentioned in a discussion about alternative GCs for CoreCLR:
https://github.com/dotnet/coreclr/issues/430
In any case, feel free to open a specific issue in the CoreCLR GitHub repository about this. Implementing an entirely new GC is a lot of work but simply improving the existing one might be another story. Though the fact that it requires kernel
support doesn't help, it may be done but it will have to wait until the kernel guys implement support for it and that will take time. There may also be legal problems if someone has a patent for such a technique.
Regarding performance: modifying page tables has a certain cost. It's possible to find that for smaller blocks it's faster to just copy the block than to modify the page tables. An implementation would probably have to employ a mix of memory copying and
page table modification and in that case you don't even need to require that objects are allocated on a page boundary. Keep in mind that it's not required to deal with one object at a time, there may be consecutive live objects that can be copied in one go.
As for Joel ramblings, well:
"I have a master in computer science and know the speed of the algorithm of compacting a large heap."
Given that you have a history of misleading/inaccurate/confusing claims such a statement is priceless. I suppose that during your master courses you were never taught basic data structures and as result you don't know that hashtable searches are O(1)
and that sorting does help with searching:
https://social.msdn.microsoft.com/Forums/en-US/09a3116a-9626-401f-8c45-c0e8b09da69c/fastest-search-algorithm?forum=csharpgeneral
In this particular case you seem to confuse the term heap as used in GC with the heap data structure. Great.
Thank you! I have opened a specific issue in the CoreCLR GitHub repository about this: https://github.com/dotnet/coreclr/issues/555

Similar Messages

Best way to compress large video files

How do i compress large video files so they dont take upso much space on my imac.

If you want to retain the quality you don't.
What format are they in now?

Serializing large objects

Dear All,
I am trying to serialize a HashMap containing Strings as keys and and ArrayLists as values. This works fine when the number of entries is small, but does not work when the number of entries are really large, in which case an java.lang.OutOfMemoryError: Java heap space is thrown. This happens when the number of entries is about 50,000 and the total number of objects stored in the lists exceed 500,000.
Some posts suggest that this should have been fixed in Java 1.6 but it still doesn't seem to work. Do I need to provide the "-mx" flag to get this to work, or is there another way to serialize large objects? Any thoughts welcome.
Cheers,
Fred

How about implementing custom serialization (by implementing Externalizable), and writing the values as zippied entries. As keys are strings, they can be easily zipped and the array list values should be zipped. Is it possible? ...am I crazy????. Just give it a try :-)

Compressing large folders containing video files

Is there any way to compress large folders (600GB) containing video files? I have footage on my internal 1TB drive which i need to get to a client, who does not have a big enough drive but needs the footage urgently.

BanditBiker wrote:
And then what... How can I select them and copy the directory containing that VIDEO_TS folder to another folder ?
If you search for Video_TS, then the smart folder will contain links to those folders. Just copy them.
When you click on the results of that smart search, Finder will open up with all those files showing.
They are already selected.

What is the best way of compressing a large 3 hour final cut file

What is the best way of compressing a large 3 hour final cut file. I shot the play and it is in final cut and I rndered it so now I have a 22gb file that I need to put on a dvd . Any suggestions
Thanks
Macbook Pro
2.3 GHz Intel Core i7
with Final Cut Pro 7.0.3
using Lion as operating system

Presuming your menus aren't complicated(include audio or video) the total size for the MPG-2 and AC3 files should probably be under 8GB for a dual layer disk. The inspector will tell you the estimated size. 2-pass varible bit rate would be recommended.
Trying to fit 3 hours on a DVD-5 will only bring very noticable quality hits. Compressor will let you change the average bit rate so that you can fit 174 minutes but trade-off isn't worth it in my opinion.
Be aware that dual-layer -R and +R media may not play well for everyone everywhere.
I presume you are not making 1000 or more copies? If you were replication could solve this.
One other alternative would be to break-up the show into two parts and spread it across two DVD-5s.

Too large java heap error while starting the domain.Help me please..

I am using weblogic 10.2,after creating the domain, while starting the domain,I am getting this error.Can anyone help me.Please treat this as urgent request..
<Oct 10, 2009 4:09:24 PM> <Info> <NodeManager> <Server output log file is "/nfs/appl/XXXXX/weblogic/XXXXX_admin/servers/XXXXX_admin_server/logs/XXXXX_admin_server.out">
[ERROR] Too large java heap setting.
Try to reduce the Java heap size using -Xmx:<size> (e.g. "-Xmx128m").
You can also try to free low memory by disabling
compressed references, -XXcompressedRefs=false.
Could not create the Java virtual machine.
<Oct 10, 2009 4:09:25 PM> <Debug> <NodeManager> <Waiting for the process to die: 29643>
<Oct 10, 2009 4:09:25 PM> <Info> <NodeManager> <Server failed during startup so will not be restarted>
<Oct 10, 2009 4:09:25 PM> <Debug> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

Thanks Kevin.
Let me try that.
Already more than 8 domains were successfully created and running fine.Now the newly created domain have this problem.I need 1GB for my domain.Is there any way to do this?

Large JVM heap sizes under windows xp?

I have a windows XP Pro SP1 box with 2 GB of physical RAM. I am using the latest jdk, 1.4.2_05.
I understand that 32 bit windows by default allows 2 GB address space for each application, permanently reserving 2 GB for the OS. (There are also apparently special builds going back to win2k which have a special mode where applications can address 3 GB and the OS addresses 1 GB; if anyone knows how to turn this mode on under win xp please let me know.)
So, I should be able to allocate ~2 GB to the JVM (the number is slightly less than 2 GB due to overhead associated with each process in windows) using something like
     -Xms1900m -Xmx1900m
as command line args to the java command.
But when I try the above it crashes with the error
     Error occurred during initialization of VM
     Could not reserve enough space for object heap
Experimentally, I found that the maximum value which works on my box is
     -Xms1200m -Xmx1200m
which seems way smaller than it ought to be.
Doing a forum search, I found others who report similar issues, e.g.
     http://forum.java.sun.com/thread.jsp?forum=136&thread=522506
But I did not find a decent solution or up to date discusssion.
Does anyone know with windows xp pro what the jvm memory limit is?
As part of my sun search, I came across the following old (circa 2000) but interesting bug report:
     http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4358809
I have never heard of the rebase program before. Is it built into windows? Is it safe to use? Anyone ever try the command as described there?

In addition to this forum posting, I got in email communication with Peter Kessler, the technical lead for the garbage collector in the HotSpot(TM) virtual machine). Below are snippets from some of his emails to me.
#1:
I haven't tried pushing Windohs XP to see how big I could get the
heap. I'm pretty sure you don't want to get try to get it to 1.9GB,
since that wouldn't leave much room for DLL's, JVM data structures
outside the heap, thread stacks, I/O buffers, etc.
I don't know what the library C:\WINDOWS\System32\LPK.DLL is, or if
you could successfully rebase it closer to 0x76000000. The issue
with rebasing DLL's is that people have sometimes given thought to
where they should be, given other libraries they are typically used
with. (If a DLL can't be loaded at its default base address, it
gets relocated dynamically. That slows down program startup; and
often means that the library can't be shared with other applications
running on the same machine, increasing your need for swap file
space.)
It does seem weird that we would make the default base for jvm.dll
be 0x08000000, but the base of hpi.dll at 0x10000000. [See email #3 below]
The jvm.dll isn't going to get that much larger any time soon. I'll ask around
about that one. (The library bases for JDK-1.4.2 are different
from those reported in bug id 4358809 for because we periodically
reconsider where to base the libraries, given the other libraries
on a particular release of Windohs.)
So, I don't have any great advice on how to squeeze another 400MB of
heap out of your machine. I suppose it's fruitless to ask if you
could squeeze 400MB of data structures out of your application?
Have you run an allocation profiler on it?
Since you are willing to switch operating systems, you can download
a copy of Solaris 10 for x86 platforms (for free) from
http://wwws.sun.com/software/solaris/10/
I just tried a Solaris 10 x86 box, and could get a -Xmx2900m jvm to
start. But without your application it's hard to know if it would
run well with that size heap (e.g., if it would have room for thread
stacks, file buffers, etc.), so your milage may vary, but 2.9GB is
larger than you say you need. There are lots of other benefits of
running Solaris. I don't know about making you a poster child, but
I could ask about that, too.
#2:
I'm told that some of the popular distibutions of Linux ship with
a "4GB" kernel, which allows one to get 3GB (2.4 kernels?) or almost
4GB (2.6 kernels?) of address space for user processes. I haven't
used those kernels myself. Among the reasons the 4GB kernel isn't
the standard kernel is that some user programs are surprised (in a
bad way) by finding their code or data in the "negative" part of
the address space, or crossing the "sign-bit" boundary. (Of course,
all addressing arithmetic should be done as unsigned quantities!)
Our Java virtual machine shouldn't have those problems, so that
kernel might work for you.
#3:
Ah. It looks like in JDK-1.4.2 (and earlier) we forgot to
explicitly set the base for hpi.dll, so it got the default
base of 0x10000000. (Don't we love learning these details
about Windohs?) As I said, we periodically reconsider the
bases for the libraries. We seem to have done that, more
thoroughly, for JDK-1.5.0:
Base Size Version Path
0x00400000 0xc000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\bin\java.EXE
0x77f80000 0x7d000 5.00.2195.6899 C:\WINNT\system32\ntdll.dll
0x7c2d0000 0x62000 5.00.2195.6876 C:\WINNT\system32\ADVAPI32.dll
0x7c570000 0xb8000 5.00.2195.6897 C:\WINNT\system32\KERNEL32.DLL
0x77d30000 0x71000 5.00.2195.6904 C:\WINNT\system32\RPCRT4.DLL
0x78000000 0x45000 6.01.9844.0000 C:\WINNT\system32\MSVCRT.dll
0x6d6b0000 0x185000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\jre\bin\client\jvm.dll
0x77e10000 0x65000 5.00.2195.6897 C:\WINNT\system32\USER32.dll
0x77f40000 0x3e000 5.00.2195.6898 C:\WINNT\system32\GDI32.DLL
0x77570000 0x30000 5.00.2161.0001 C:\WINNT\system32\WINMM.dll
0x6d2f0000 0x8000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\jre\bin\hpi.dll
0x690a0000 0xb000 5.00.2134.0001 C:\WINNT\system32\PSAPI.DLL
0x6d680000 0xc000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\jre\bin\verify.dll
0x6d370000 0x1d000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\jre\bin\java.dll
0x6d6a0000 0xf000 1.05.0000.0000 C:\pbk\deployed\JDK-1.5.0\jre\bin\zip.dll
You might want to try the "release candidate" (i.e., what we
hope will be the final bits) of JDK-1.5.0 from
http://java.sun.com/j2se/1.5.0/download.jsp
and see if it helps.

An effective way for JSP/servlets to interact with databases

Hi
I was wondering if anyone has an effective way of connecting to databases using javabeans and JSP.
I am new to all this JSP stuff and would appreciate some help
cheers

hai,
it is better if u go for javaBeans
for connection purpose
as this is reused many times
and u can avoid large stuff of code
in the jsp.Also try connection pooling pooling if
user name to db for most user is same !!!!
sample code for database bean
import java.sql.*;
import java.io.*;
* This class determines the database connectivity by defining the database url,
* database driver and database connection and other functions to prepare & execute
* sql queries.
public class DBBean {
String dbURL; //database url
String dbDriver = "sun.jdbc.odbc.JdbcOdbcDriver"; //database driver
private Connection dbCon; //database connection
private PreparedStatement pre;
private Statement s ;
boolean prepareflag=false;
public IntranetBean() {
super();
public boolean connect() throws ClassNotFoundException, SQLException {
//install the driver
Class.forName(dbDriver);
//opens a connection
dbCon = DriverManager.getConnection("jdbc:odbc:ureodbc");
return true;
public void close() throws SQLException {
//close the statement
if(prepareflag){pre.close();}
else{
s.close();}
//close the connection
dbCon.close();
prepareflag=false;
public ResultSet execSQL(String sql) throws SQLException {
//gives a statement object
s = dbCon.createStatement();
//execute the sql command
ResultSet r = s.executeQuery(sql);
return (r == null) ? null : r;
public void execUPDATE(String sql) throws SQLException {
//gives a statement object
s = dbCon.createStatement();
//execute the sql command
s.executeUpdate(sql);
public boolean prepareSQL(String sql) throws SQLException {
//gives a prepared statement object
pre = dbCon.prepareStatement(sql);
prepareflag=true;
return true;
public boolean execpreparedSQL(String sql) throws SQLException {
//execute the sql command
pre.execute(sql);
return true;
public ResultSet getpreparedResultSet() throws SQLException {
//get resultset
ResultSet r = pre.getResultSet();
return (r == null) ? null : r;
public void setInt(int i,int no) throws SQLException {
pre.setInt(i,no);
public void setDouble(int i,double dbl) throws SQLException {
pre.setDouble(i,dbl);
public void setString(int i,String str) throws SQLException {
pre.setString(i,str);
public void setDate(int i,Date d) throws SQLException {
pre.setDate(i,d);
public void setTime(int i,Time tm) throws SQLException {
pre.setTime(i,tm);
public void setNull(int i,int type) throws SQLException {
pre.setNull(i,type);
}

PDF File Size - any way to compress further?

We have are using the Crystal for .NET export method to export a report to a .pdf file, and are having an issue with the pdf file size on a report that contains images. The images are stored in a SQL2005 database as blob or varbinary(max). The report executes a stored procedure that selects data (including images) to produce a quotation. There are input parms to decide which images to print (for example A, B or both A&B). Both types of images can appear at the line level on the quotation. Type A images print in the main report, Type B images are in a subreport. Both the main report & subreport execute a stored procedure to select an image type.
Our issue, when both Type A and B image is selected to print on a 293 line quote. The PDF file size is 44.23MB
When Type A only is selected, the PDF file size is 2.64MB
When Type B only is selected, the PFD file size is 43.95MB.
There are more Type B images that would print at the line level than Type A, but is there any way to compress this down further as it is too large to email.

You mention that you are using Crystal for .NET, but not what version; CR for .NET 2003, 2005, 2008. 2010?
Applying the latest fixes for the correct version of CR would be the first thing to do.
Next, I'd have a close look at image B as it appears to be the one adding the most "bulk". How is this image different from image A?
General tips re. images:
Save the files a BMPs.
When .jpg image is inserted into Crystal Reports it is converted into bitmap format and as the result of the conversion the report may loose some quality, scale and other issues may arise... E.g.; Crystal Reports is not so good a graphics management tool
When an image is saved as a .bmp, then there is no conversion required and thus the quality of the image should be preserved.
Best practice: save the image with high resolution and required size as a .bmp format then insert this image into Crystal Reports.
Resize the image to the smallest possible size and downgrade the DPI to 72. This will ensure your image is as small as possible and Crystal Reports will have to put in the least amount of work to display it.
Have a look ar KB [1241630 - Exporting a Crystal report (XI) to PDF generates a big PDF file|http://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/com.sap.km.cm.docs/oss_notes_boj/sdn_oss_boj_bi/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/scn_bosap/notes%7B6163636573733d36393736354636443646363436353344333933393338323636393736354637333631373036453646373436353733354636453735364436323635373233443330333033303331333233343331333633333330%7D.do]. This KB, may apply to your version of CR, or not. In any case, you will have to use the KB as a guide and determine what the appropriate registry entry would be for your version of CR.
One more thing. Many people like to use jpg files as they are smaller than bmp files. However, as far as Crystal Reports is concerned, this is inconsequential. The report file will be the same size if a file is inserted as a jpg or a bmp. This is due to the jpg conversion to bmp Crystal Reports does internally.
Ludek
Follow us on Twitter http://twitter.com/SAPCRNetSup
Got Enhancement ideas? Try the [SAP Idea Place|https://ideas.sap.com/community/products_and_solutions/crystalreports]

Any way to increase the default Heap size for all Java VMs in Solaris 8

Hello,
I have a java product that deals with large databases under Solaris 8. It is a jar file, started by a cron job every night. Some nights it will fail because it runs out of Heap memory depending on the amount of records it has to deal with. I know that I could increase the java VM heap size with "java -jar -mx YY JARFILE" command but I have other java products that are showing the same behavior, and I would like to correct them all in one shot if possible.
What I would like to do is find a system or configuration parameter that forces all Java VMs to use a larger MAX Heap size than the default 16M specified in the Man page for Java. Is there a way to accomplish that?
TIA
Maizo

You could always download the source and modify it.

How can I create an iridescent effect in a Photoshop 3D object?

How can I create an iridescent effect in a Photoshop 3D object? I have searched everywhere for downloadable option for a mac with no luck. I'm working with CS6.

NO way. Such stuff isn't even commonly available in many 3D programs becaus it's actually pretty advanced shader stuff...
Mylenium

Is there a way to compress the size of an iMovie to be able to send it in an email?, is there a way to compress the size of an iMovie to be able to send it in an email?, is there a way to compress the size of an iMovie to be able to send it in an email?

so i made a slideshow for someone and i cant email it to them bec the file is to big... is there a way to compress it?

Hi
Yes and No
Yes
• Share as QuickTime
• Convert it to 3gp
No
• Will still be larger than most e-mail account's accepts - Mine are limited to 25Mb max
• Quality will be worse than bad
I use
• YouSentIt
Down load movie
Get an internet address
Send address to reciever
I think it was free for 100-200Mb
And by using H264 and small size I got an usably movie over.
Yours Bengt W

Displaying an image in large object type

hello
I am getting large object ( image) from a database.
like this way:
obj.read(buf, 0, obj.size());
I was not able to this image by OutputStream object
could you give me a hint ?
regards
Ahmet Temiz

orkun wrote:
InputStream is=rs.getBinaryStream(1); So you get an InputStream to read the image from.
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];And you create a byte array which contains a bunch of zeros.
for (int length; (length = is.read(buffer)) != -1;) {
out.write(buffer, 0, length); # out is in OutputStream
}And then you ignore the InputStream where the image was, and copy all the zeroes from
the byte array to the output.
that did not display any image.
what may the problem be ?Do you need more explanation?

HELP - With Object Heap

Hello,
I was wondering if you could help me with my problem. I have recently designed a website that uses the JVM on the server side (http://amazingestate.com) on a shared linux machine. The site was working fine until the owners of the server installed a few new files and I now get errors when running the site.
The java classes run fine from a command prompt but when are called by a cgi I get a �could not reserve enough space for the object heap� error. The java programs are small and don�t require large memory requirements.
I was thinking the error is within the shell script in terms of allocating space for the program. I am not sure how to allocate memory within the shell account at runtime within the cgi wrapper script. The script bellow used to work fine and also run fine when telneting into my account. The error only occurs when running inside a cgi request.
I have also tried the �Xmx and �Xms commands without luck.
I have also tried to test the ulimit which is also set on unlimited.
The following script can be found at
http://amazingestate.com/cgi-bin/publicView/RND10.cgi
which allows the viewing of the error
#!/bin/sh
echo "content-type:text/html"
echo
ulimit
echo "<HTML><BODY>"
echo "</BODY></HTML>"
/usr/java/j2sdk1.4.0/bin/java -server -cp /home/amazing/ss/publicView RND10
Anyones help would be greately appreciated. Thanks for your time.
Dale Miller

You're going to start a new JVM every time you get a page hit? Are you aware of how costly this is (slow and memory-intensive)? This is exactly the type of requirement that Java servlets and server pages (jsps) are designed to take care of. You should consider using these.
Regarding your specific problem, the only thing I could recommend trying are the -Xmx/ms switches, which you seem to have tried.

What is the most effective way to write Statement to catch and reverse errors during query excution?

Hello my friends:
I am wondering what is the most effective way to deal with errors, specifically in a
Stored Procedure.
I wrote something like this:
BEGIN TRY
BEGIN TRANSACTION
/*My statements goes in here*/
IF ERROR_NUMBER() = 0 -- Do I need this line?
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
IF ERROR_NUMBER() > 0 --Do I need this line?
ROLLBACK TRANSACTION;
END CATCH;
It would make sense using the if Statement when attempting to log errors.
Just too many variations.
Thanks

Also read this great article
http://sqlblog.com/blogs/alexander_kuznetsov/archive/2009/05/13/your-try-block-may-fail-and-your-catch-block-may-be-bypassed.aspx
Best Regards,Uri Dimant SQL Server MVP,
http://sqlblog.com/blogs/uri_dimant/
MS SQL optimization: MS SQL Development and Optimization
MS SQL Consulting:
Large scale of database and data cleansing
Remote DBA Services:
Improves MS SQL Database Performance
SQL Server Integration Services:
Business Intelligence

An effective way to compress Large Object Heap

Similar Messages

Maybe you are looking for