External memory allocation and management using C / LabVIEW 8.20 poor scalability

Question

Hi,
I have multiple C functions that I need to interface. I need
to support numeric scalars, strings and booleans and 1-4 dimensional
arrays of these. The programming problem I try to avoid is that I have
multiple different functions in my DLLs that all take as an input or
return all these datatypes. Now I can create a polymorphic interface
for all these functions, but I end-up having about 100 interface VIs
for each of my C function. This was still somehow acceptable in LabVIEW
8.0 but in LabVIEW 8.2 all these polymorphic VIs in my LVOOP project
gets read into memory at project open. So I have close to 1000 VIs read into memory when ever I open my project. It takes now about ten minutes to
open the project and some 150 MB of memory is consumed instantly. I
still need to expand my C interface library and LabVIEW doesn't simply
scale up to meet the needs of my project anymore.
I now
reserve my LabVIEW datatypes using DSNewHandle and DSNewPtr functions.
I then initialize the allocated memory blocks correctly and return the
handles to LabVIEW. LabVIEW complier interprets Call Library Function
Node terminals of my memory block as a specific data type.
So
what I thought was following. I don't want LabVIEW compiler to
interpret the data type at compile time. What I want to do is to return
a handle to the memory structure together with some metadata describing
the data type. Then all of my many functions would return this kind of
handle. Let's call this a data handle. Then I can later convert this
handle into a real datatype either by typecasting it somehow or by
passing it back to C code and expecting a certain type as a return.
This way I can reduce the number of needed interface VIs close to 100
which is still acceptable (i.e. LabVIEW 8.2 doesn't freeze).
So
I practically need a similar functionality as variant has. I cannot use
variants, since I need to avoid making memory copies and when I convert
to and from variant, my memory consumption increases to three fold. I
handle arrays that consume almos all available memory and I cannot
accept that memory is consumed ineffectively.
The question is,
can I use DSNewPtr and DSNewHandle functions to reserve a memory block
but not to return a LabVIEW structure of that size. Does LabVIEW
carbage collection automatically decide to dispose my block if I don't
correctly return it from my C immediately but only later at next call
to C code. Can I typecast a 1D U8 array to array of any dimensionality and any numeric data type without memory copy (i.e. does typecast work the way it works in C)?
If I cannot find a solution with this LabVIEW 8.20 scalability issue, I have to really consider transferring our project from LabVIEW to some other development environent like C++ or some of the .NET languages.
Regards,
Tomi
Tomi Maila

Answer 1

I have to answer to myself since nobody else has yet answered me. I came up with one solution that relies on LabVIEW queues. Queues of different type are all referred the same way and can also be typecased from one type to another. This means that one can use single element queues as a kind of variant data type, which is quite safe. However, one copy of the data is made when you enqueue and dequeue the data.
See the attached image for details.
Tomi Maila
Attachments:
variant.PNG ‏9 KB