CFS and memory mapped file

I would like to know if it is possible to memory map (mmap) a file that is residing on a cluster file system (CFS or GFS).
If I remember correctly, memory mapping a file residing on NFS has issues.
Thanks,
Harsh

I'm using SC 3.1u4 on Solaris 9. I ran in to a problem with memory mapped files on CFS.
I've multiple processes (on one cluster node) sharing such a file that was created using the following command:
mmap ((caddr_t)0,SOME_SIZE,(PROT_READ | PROT_WRITE), (MAP_SHARED | MAP_NORESERVE),fd,0);
Issuing msync with MS_INVALIDATE as the third argument is ok. But when some other process tries to read the memory the node seems to hang.
I can't examine the processes using pstack or truss as both of them get hung too. Only way out of this mess is to reboot the node.
I can't imagine this problem hasn't been seen before. Is there a patch for it?

Similar Messages

Nio ByteBuffer and memory-mapped file size limitation

I have a question/issue regarding ByteBuffer and memory-mapped file size limitations. I recently started using NIO FileChannels and ByteBuffers to store and process buffers of binary data. Until now, the maximum individual ByteBuffer/memory-mapped file size I have needed to process was around 80MB.
However, I need to now begin processing larger buffers of binary data from a new source. Initial testing with buffer sizes above 100MB result in IOExceptions (java.lang.OutOfMemoryError: Map failed).
I am using 32bit Windows XP; 2GB of memory (typically 1.3 to 1.5GB free); Java version 1.6.0_03; with -Xmx set to 1280m. Decreasing the Java heap max size down 768m does result in the ability to memory map larger buffers to files, but never bigger than roughly 500MB. However, the application that uses this code contains other components that require the -xMx option to be set to 1280.
The following simple code segment executed by itself will produce the IOException for me when executed using -Xmx1280m. If I use -Xmx768m, I can increase the buffer size up to around 300MB, but never to a size that I would think I could map.
try
String mapFile = "C:/temp/" + UUID.randomUUID().toString() + ".tmp";
FileChannel rwChan = new RandomAccessFile( mapFile, "rw").getChannel();
ByteBuffer byteBuffer = rwChan.map( FileChannel.MapMode.READ_WRITE,
0, 100000000 );
rwChan.close();
catch( Exception e )
e.printStackTrace();
I am hoping that someone can shed some light on the factors that affect the amount of data that may be memory mapped to/in a file at one time. I have investigated this for some time now and based on my understanding of how memory mapped files are supposed to work, I would think that I could map ByteBuffers to files larger than 500MB. I believe that address space plays a role, but I admittedly am no OS address space expert.
Thanks in advance for any input.
Regards- KJ

See the workaround in http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038

Error code 1450 - memory mapped file

Hello,
in my application I am using memory mapped files. I have three of it, the maximum size of the biggest one is 5MB. I store 64 Waveforms from a DAQ card in it.
The application runs fine, but sometimes comes an error, when I try to access the MMF. The error code is 1450, "insufficient system resources"
Is a size of 5MB too big? Should I rather create one MMF for each waveform?

Hi mitulatbati,
which development tools are you actually using?
Which platform, libraries and so on...?
Can you post example code?
Marco Brauner NIG

Memory-mapped file is possible?

Hi everyone, I'm a new Labview user and I want to start a new project that uses Memory mapped file.
I have a working C# code to read the $gtr2$ MMF, where i simple use
MemoryMappedFile.OpenExisting("$gtr2$")
to get data from it.
How it is possible to read this kind of file in labview? I can't find anything useful on the web.
I'm using a LabVIEW 2013 student edition.
Thanks to everyone who wants to answer my question.
Have a nice day.

Hi,
I too only have done the CLAD…
You have to look for DotNet examples, you will find them here in the forum…
And usually it helps to read the documentation for that MMF class to recreate your C++ code in LabVIEW!
Best regards,
GerdW
CLAD, using 2009SP1 + LV2011SP1 + LV2014SP1 on WinXP+Win7+cRIO
Kudos are welcome

Memory mapped files Are they still used.

To System programmers.
In some of my old code David used memory mapped files in handling huge sets of random points. The code reads in the whole file and then sets flags similar to an async process. The filemapping handles memory instead of using mallocs. the
data maybe stored on the heap or in hte global stack. I went back to Viusal Studio 6 and tried to take out the code as the standard c++ handles a full file read as a char buffer as a void * structure. I found some valloc data types and
then found the newer filemapping routines in VS2013. Plus an explanation of global stack and heap.
Are software developers using file mapping or are they using say vectors to form stl structures.
Cheers
John Keays
John Keays

Here is some typical code in the old C. This is close to the code I used in Visual studio 6. I need to put this in vs2013 under c++ or C++ 11/14. I have guessed the file handle open and size code.
main{
int fsize, numRecords;
Point *allPoints;
fsize = readAllFile(name, &allPoints);
numRecords = fsize/ sizeof(Point);
for (i=0; i < numRecords:; I++) printf("rec %d values x %.3f\n", i, allPoints[i].x);
int
readAllFile(char*name, void **addr){
file *fh;
int fsize;
openf(fh, name);
fsize = getfilesize(fh);
*addr = malloc(sizeof(char)*fsize);
fclose(fh);
return fsize;
This is the boilerplate for the file reads. Even tried this with text files and parsing the text records. Instead of the mallocs you suggest vector and the scheme of the code remains the same.
For a lidar file the xyz records have grown from 10,000 in the 1990's to 1,000,000 points in the mid 2000's. For this size file 24 M bytes are allocated in one hit. The whole of the Gold Coast in terms of lidar points in 2003 was 110 million
points. It could be more.
Where is the data stored in the Malloc, Vector or memory Mapped file. What is good and bad practice.
Cheers
john Keays
John Keays

Memory mapped files

Does anyone know if there is any way to use memory mapped files in Java? If so, what are the calls that emulate the C++ calls to CreateFileMapping() MapViewOfFile() and OpenFileMapping()?

http://java.sun.com/j2se/1.4.1/docs/api/java/nio/MappedByteBuffer.html

How to truncate a memory mapped file

If one maps a file, the mapped size will become the file size. So the size parameter passed to the map() method of FileChannel should be carefully calculated. However, what if one can't decide beforehand the size of the file?
I tried to use truncate(), but that throws a runtime exception: truncate() can't be used on a file with user-mapped section open.
public class MapFileSizeTest extends TestCase
public void testMapFileSize() throws Exception
    final File file=new File("testMapFileSize.data");
    FileChannel configChannel= new RandomAccessFile(file, "rw").getChannel();
    //this will result a file with size 2,097,152kb
    MappedByteBuffer configBuffer= configChannel.map(FileChannel.MapMode.READ_WRITE,
        0, 1000000000);
    configBuffer.flip();
    configBuffer.force();
    //truncate can't be used on a file with user-mapped section open
//    configChannel.truncate( configBuffer.limit());
    configChannel.close();
}Could somebody please give some suggestions? Thank you very much.

The region (position/size) that you pass to the map method should be contained in the file. The spec includes this statement: "The behavior of this method when the requested region is not completely contained within this channel's file is unspecified. " In the Sun implementation, we attempt to extend the file if the requested region is not completely contained but this is not required by the specification. Once you map a region you should not attempt to truncate the file as it can lead to unspecified exceptions (see MappedByteBuffer specification). Windows prevents it; others allows it but cause access to the inaccessible region to SIGSEGV (which must be handled and converted into a runtime error).

High Page Reads/Sec on Windows 2008 R2 64-bit running on VMware but very low Real Memory & Page file Usage.

Hello All,
Below is the server configuration,
OS: Windows 2008 R2 Enterprise 64 Bit
Version: 6.1.7601 Service Pack 1 Build 7601
CPU: 4 (@ 2.93 GHz, 1 core)
Memory: 12 GB
Page file: 12 GB
1. The actual utilization, be it a 15 minute sample, hourly, weekly etc, the utilization of real memory has never crossed 20% and the page file usage is at 0.1%. For some reason, the Pages/Sec>Limit% counter reports 100% continuously regardless of the
sampling intervals. Upon further observation, the Page Reads/Sec value is somewhere between 150~450 and Page Input/Sec is somewhere between 800~8000. Does this indicate a performance bottleneck? (I've in the interim asked the Users, App. Owners to see if they
notice any performance degradation and awaiting response). If this indicates a performance issue, please could someone help list down how to track this down further to which process/memory mapped file is causing it? and what I should go about performing to
fix this problem please?
p.s., initially the Security logs were full on this server and since page file is tied to Application, Security and System logs, this was freed up to see if this is causing the high page reads but this doesn't.
2. If the above does not necessarily indicate a performance problem, please can someone reference few KB articles that confirms this? Also, in this case, will there be any adverse effects if attempting to fine tune a server which is already running fine?
assuming App. Owners confirm there isn't any performance degradation.
Thanks in advance.

Hi,
Based on the description, we can try to download Server Performance Advisor (SPA) to help further analyze the performance of the server. SPA can generate comprehensive diagnostic reports and charts and provides recommendations to help you quickly analyze
issues and develop corrective actions.
Regarding this tool, the following articles can be referred to for more information.
Microsoft Server Performance Advisor
https://msdn.microsoft.com/en-us/library/windows/hardware/dn481522.aspx
Server Performance Advisor (SPA) 3.0
http://blogs.technet.com/b/windowsserver/archive/2013/03/11/server-performance-advisor-spa-3-0.aspx
Best regards,
Frank Shen
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected]

Memory Mapping on a Virtual Machine

I have an application that memory maps a number of files. Occasionally it needs to unmap them so that they can be updated. On a real machine this process works flawlessly but I have two installs that are on virtual machines. On the virtual machine installs the unmapping occasionally seems to either fail or at least take a long time. When I come to update the file I get the "The requested operation cannot be performed on a file with a user-mapped section open" error message.
The failure is fairly random, sometimes the update process works sometimes is fails and it doesn't consistently fail on the same mapped file which is why I think it's something to do with the timing of the unmapping and the virtual machine environment. Both virtual machine installs are running on Windows 2003 Server.
Has anyone else seen this? I'm going to try inserting a pause between the unmapping and the update but that feels like a hack, I'd rather a call back to tell me the unmapping is complete but I don't suppose that's possible.

the_doozer wrote:
Ok, I'll grant you there is no way to explicitly unmap a mapped file in Java (a huge failing of the file mapping system IMHO) but closing any open FileChannel and nulling out the MappedByteBuffer is, in my experience, normally enough to cause the OS to unmap the file. This system lets me update files quite happily on all but the virtual machine system.I think you've been lucky in that case. I had some test cases that consistently failed since I couldn't delete the memory mapped file that I previously had created.

How can I generate a map file with LabVIEW?

We wish to use a product which inserts code into our executable to prevent tampering with it by crackers. The program, however, takes the executable file, as well as the map file (which is commonly generated by c++ compilers) and uses the map file to determine where in the exe the critical routines that need protection are at. I can not, however, determine how to create such a map file for a LabVIEW generated executable. Is there a special build option I need to invoke?

Yes, I'm familar with NI's licensing technology, having talked with someone (you, Dennis, I believe) about it before. The problem we have is our software is sold to factories in China where there are no internet connections. We have a physical key 'dongle' which must be present in order for the executable to be willing to run. However, it appears that people are taking the executable which LabVIEW creates and they are editing it, probably by using a a debugger and tracing to the the code which checks for the dongles presence and bypassing it. To my knoweledge, NI's products don't do anything to prevent this, right?
We found a company which sells a product that encryptes, checksums, etc... an executable file, but it needs to know the layout of functions in the exe in order to determine which areas to focus the obfuscation on. They were sort of matter of fact when they said it needs the exe and the map file, as if they expected any language which produced an exe could produce a map file.

XSLT and Java Mapping

Hello friends,
Can any one know that how to create a XSLT and Java Mapping file(Jar File).
I am not able to find any of the example so if you create a example or find some where please let me know on that How to create such files...
Thanks...
Gaurav Jain

Hi,
I am trying to invoke a Java Extension Function from a XSLT. I added both the class file and the XSLT file to a folder, then zipped the entire folder and uploaded into the Archive for my Interface MApping.
The CLASS gets loaded but the runtime engine gives an error when calling the JAVA Function.
Here is what it says:
Creating mapping sequence with 2 steps. --- Creating Java mapping Deliveries/stringHandler --- Creating mapping sequence with 2 steps. --- Creating XSLT mapping Deliveries/test1 --- Using MappingResolver with context URL /sapmnt/XD1/global/xi/mapping/http%3A%2F%2Fwolterskluwer%2Ecom%2Fxi%2F wk_midas_deliveries%2Ftest/cd2e6820267511d9cb52c3c5143c3524/ --- Loaded class Deliveries.stringHandler --- com.sap.aii.ibrun.server.map.MappingRuntimeException: at com.sap.aii.ibrun.server.map.MappingRuntimeException.code_STYLESHEET_OR_CLASS_NOT_FOUND(MappingRuntimeException.java:91) at com.sap.aii.ibrun.server.map.RUMappingJava.instantiate(RUMappingJava.java:168) at com.sap.aii.ibrun.server.map.RUMappingJava.execute(RUMappingJava.java:41) at com.sap.aii.ibrun.server.map.RUSequence.execute(RUSequence.java:50) at com.sap.aii.ibrun.server.map.RURunner.run(RURunner.java:58) at com.sap.aii.ibrun.server.map.RUManager.run(RUManager.java:66) at com.sap.aii.ibrun.sbeans.map.MappingRequestHandler.handleRequest(MappingRequestHandler.java:67) at com.sap.aii.ibrun.sbeans.map.MappingServiceImpl.processFunction(MappingServiceImpl.java:83) at com.sap.aii.ibrun.sbeans.map.XMappingServiceObjectImpl0.processFunction(XMappingServiceObjectImpl0.java:24) at com.sap.aii.ibrun.sbeans.map.MappingServiceKey.processFunction(MappingServiceKey.java:10) at java.lang.reflect.Method.invoke(Native Method) at com.inqmy.services.rfcengine.RFCDefaultRequestHandler.invokeBean(RFCDefaultRequestHandler.java:83) at com.inqmy.services.rfcengine.RFCDefaultRequestHandler.handleRequest(RFCDefaultRequestHandler.java:50) at com.inqmy.services.rfcengine.RFCJCOServer.handleRequest(RFCJCOServer.java:69) at com.sap.mw.jco.JCO$Server.dispatchRequest(Unknown Source) at com.sap.mw.jco.rfc.MiddlewareRFC$Server.nativeListen(Native Method) at com.sap.mw.jco.rfc.MiddlewareRFC$Server.listen(Unknown Source) at com.sap.mw.jco.JCO$Server.listen(Unknown Source) at com.sap.mw.jco.JCO$Server.run(Unknown Source) at java.lang.Thread.run(Thread.java:513) --
Please tell me what is the problem?
Thanks for your time
A

Importing Map Files into a Project

Hello,
The programmers at my job create map files, and when map
files are sent to me, they're sent in a format that includes hex
values (e.g.: 0x1177E). I've noticed map files that were imported
in other help projects have this format:
#define ID_Size 1001
Out of curiousity, do map files have to be in the above
format before importing them into a help project, or can RoboHelp
perform a conversion to the above format once map files are
imported?
Since I receive map files from programmers, do they need to
convert the map files to the above format before sending them to
me?
I found something in the Knowledge Base Guided Search on
this, but I want to know if anyone knows of a new process that may
be in the latest version (X5) of RoboHelp.
Thanks,
Felicia

Hi John,
Thanks for your response. These are a few map IDs/map numbers
and how they appear in one of the map files given to me by a
programmer:
HID_SELECTION 0x10005
HID_SNAPS_CANCEL 0x10006
HID_TOOLBARS_PRINT 0x10007
HID_T_FITTING_MANIFOLD 0x11AB2
I wonder, according to what's stated in the Help for
RoboHelp, if the map ID should appear like this before I import the
map file into a project:
#define HID_SELECTION 0x10005
or:
#define HID_SELECTION 65541
Does the "#define" have to be in front of the map ID and map
number before the import takes place?
Felicia

FileChannel and mapping files

Hello,
I'have some strange problem with mapping file using a FileChannel and the map method.
My program creates some number of threads. These threads are accessing in a file (in a different places) to write down some information. The number of the created threads is about (max) 100 but at one moment I may have for example from 1 to 60 threads that are active.
Sometimes when a thread tries to map some region from the file, I have an exception.
So here is my code:
FileChannel fch = rafOut.getChannel();
try {
position = rafOut.getChannel().position();
//I have the exception here fch.map(....)
mbf = fch.map(FileChannel.MapMode.READ_WRITE, position, Manager.getSize());
} catch (IOException e2) {
e2.printStackTrace();
//here I'm using some putXXX() methods
//and at the and I have
mbf.force();
}This code sometimes generates this exception:
java.io.IOException: The requested operation over a mapped file was not accomplished
     at sun.nio.ch.FileChannelImpl.truncate0(Native Method)
     at sun.nio.ch.FileChannelImpl.map(Unknown Source)
     at DataWriter.run(DataWriter.java:40)
Any idea why I have this error?
Regards,
Anton
PS. The first part of the exception message was written in French, so I tried to translated it :)
Edited by: anton_tonev on Nov 5, 2007 6:49 AM

Hi ejp,
i'm trying to extend the file. In the begging the file is empty, so every thread adds data to the file.
About the mapping, yes I know that map many times the same file is not the right solution ... but for every thread I'm mapping a different part of the file, for exemple :
one thread maps from 0 byte to the 256 byte and writes there the data.
another thread maps from 256 to 1024 and so on ...
The problem is that the threads don't know the size of the data before the mapping(e.g. the future size of the entire file), if it was known, may be I could map the entire file in the memory. And there is more thing, the file sometimes becomes very big .. a hundred of Gbs, so mapping just some parts of it in the memory for me is the best solution.
Regards,
Anton

Memory mapping large files

Hi folks.
I am developing an application that has very large input files. During execution, the files will be processed twice: once, sequentially to get the position of each piece of data in the file, and then directly by seeking to a specific position to retrieve a specific piece of information.
My rational for doing this is to avoid loading the entire content of the file into memory via some data structure. However, all of the seeking/reading seems to be quite a performance hit.
Is there a way to memory map a file and then be able to read only a portion of the data based on its byteposition? I've searched around for sample code, but I can only find examples of sequential access.
Any help will be appreciated extremely!!
Thanks

That's pretty simple. Thanks
Follow-up questions:
The code I have now reads:
FileChannel fc = seqDBRAF.getChannel();
ByteBuffer roBuf = fc.map(FileChannel.MapMode.READ_ONLY, 0, fc.size());
CharBuffer cb = Charset.forName("ISO-8859-15").newDecoder().decode(roBuf);
The decode line takes a long time to execute not the "map" line. Why is this?
If/when I use the position method to "seek" to the right place, should I do this to the ByteBuffer and then decode? Or decode first and then just read from the position in the Charbuffer?
Thanks

Embedded LVM file, memory map?

I am running out of memory, "No Space in Execution Regios" when I try to built my ARM 7 project. I want to determine the code size for each of my VIs in the project. Is there a LINK file?
The build process produces an Applicaiont.lvm file that contains some (all) of the mapping mapping. I found an SDK article that showed how to read the various data/structure size and mapping format but did it not describe how the VI executibles were mapped.
What is the best way to determine the size and mapping of both data and executable code for an Embedded project build?
Solved!
Go to Solution.

There is a linker output map file located at
.../<project dir>/<proj name>/<target name>/<applicaiton name>/2.0/project/labview.map

CFS and memory mapped file

Similar Messages

Maybe you are looking for