X64 (64 bit) C API bug

Question

I am trying to get the 64 bit C API up and running.  I have the 32 bit API running and it is working correctly.  It appears that any API call in the 64 bit environment that returns more than a single data type overwrites an area of memory it doesn't own. 
I am working in VB.NET and wrapping the calls with marshaling.  The simplest example of a call that doesn't work correctly is querying the active channels on the scope:
From the C header:
niVB_Status NIVB_DECL niVB_MSO_QueryEnabledAnalogChannels(
   niVB_MSO_InstrumentHandle instrumentHandle,
   char* channels,
   size_t channelsSize,
   size_t* channelsSizeOut);
There are several different ways to code a managed to unmanaged translation for this.  One is to use an allocation to construct a pointer to a buffer area:
Pointer = Marshal.AllocHGlobal(MAX_CSTRING_LENGTH)
The declaration then looks like this:
      <DllImport(niVirtualBenchModuleName64, EntryPoint:="niVB_MSO_QueryEnabledAnalogChannels", CallingConvention:=CallingConvention.Cdecl)> _
      Public Shared Function niVB_MSO_QueryEnabledAnalogChannels_64(ByVal instrumentHandle As IntPtr, ByVal channels As IntPtr, ByVal channelsSize As UInteger, ByRef channelsSizeOut As UInteger) As niVB_Status
      End Function
When the call is made, the channelsSizeOut call returns 23 characters (which is correct).  The channels IntPtr is zeroed out however (this should never happen, it suggests the stack is getting corrupted.  If I put a huge array locally around the pointer, sometimes it's not corrupted. 
Or you can call into the function using a stringbuilder call.  It basically replaces the IntPtr parameter with a string builder using marshaling to get a pointer:
        Dim bufChannels As New StringBuilder(MAX_CSTRING_LENGTH)
then the call looks like this:
      <DllImport(niVirtualBenchModuleName64, EntryPoint:="niVB_MSO_QueryEnabledAnalogChannels", CallingConvention:=CallingConvention.Cdecl)> _
      Public Shared Function niVB_MSO_QueryEnabledAnalogChannels_64(ByVal instrumentHandle As IntPtr, <MarshalAs(UnmanagedType.LPStr)> ByVal channels As StringBuilder, ByVal channelsSize As UInteger, ByRef channelsSizeOut As UInteger) As niVB_Status
      End Function
Again, calling into this function causes the stringbuilder class to be set to (nothing) which is essentially showing that the stack space where the pointer to the stringbuilder class resides is getting corrupted.
It is important to note that this code works perfectly with the 32 bit version of the DLL (using either programming construct). 
Solved!
Go to Solution.

Answer 1

I don't think this is a C API bug; I can do the following on both x86 and x64 without triggering any of MSVC's stack canaries. (Note that for brevity there's no error-checking in this snippet.)
#include <stdlib.h>
#include <stdio.h>
#include "nivirtualbench/nivirtualbench.h"
int main(int argc, char* argv[])
niVB_LibraryHandle libHandle = NULL;
niVB_MSO_InstrumentHandle msoHandle = NULL;
size_t channelsSizeOut = 0;
char* enabledChannels = NULL;
niVB_Initialize(NIVB_LIBRARY_VERSION, &libHandle);
niVB_MSO_Initialize(libHandle, "brandonbench", false, &msoHandle);
niVB_MSO_QueryEnabledAnalogChannels(msoHandle, NULL, 0, &channelsSizeOut);
enabledChannels = (char*)calloc(channelsSizeOut, sizeof(char));
niVB_MSO_QueryEnabledAnalogChannels(msoHandle, enabledChannels, channelsSizeOut, &channelsSizeOut);
printf("Enabled channels: %s\n", enabledChannels);
free(enabledChannels);
niVB_MSO_Close(msoHandle);
niVB_Finalize(libHandle);
I suspect the problem is in your marshaling. The size_t parameters are probably best represented using System.UIntPtr, and not the UInteger type; size_t is 32-bit on 32-bit platforms and 64-bit on 64-bit platforms, but UIntegers are 32-bit on both. Trying to write channelsSizeOut is probably what's triggering the stack corruption.
Brandon Streiff
National Instruments
Software Engineer