Implementing an unmanaged C++ interface callback in C#/.Net

Ever wanted to implement a C++ interface callback in a managed C# application? Well, although that's not so hard, this is a solution that you will probably hardly find over the Internet... the most common answer you will get is that it's not possible to do it or you should use C++/CLI in order to achieve it...  In fact, in C#, you can only implement a C function delegate through the use of Marshal.GetFunctionPointerForDelegate but you won't find anything like Marshal.GetInterfacePointerFromInterface. You may wonder why do I need such a thing?

In my previous post about implementing a new DirectX fully managed API, I forgot to mention the case of interfaces callbacks. There are not so many cases in Direct3D 11 API where you need to implement a callback. You will more likely find more use-cases in audio APIs like XAudio2, but in Direct3D 11, afaik, you will only find 3 interfaces that are used for callback:

• ID3DInclude which is used by D3DCompiler API in order to provide a callback for includes while using preprocessor or compiler API (see for example D3DCompile).
• ID3DX11DataLoader and ID3DX11DataProcessor, which are used by some D3DX functions in order to perform asynchronous loading/processing of texture resources. The nice thing about C# is that those interfaces are useless, as it is much easier and trivial to directly implement them in C# instead

So I'm going to take the example of ID3DInclude, and how It has been successfully implemented for the SharpDX.

Memory layout of a C++ object implementing pure virtual methods

If you know how a C++ interface with pure methods is layout in memory, that's fairly easy to imagine how to hack C# to provide such a thing, but if you don't, here is a quick summary:

For example, the ID3DInclude C++ interface is declared like this :

// Interface declaration
DECLARE_INTERFACE(ID3DInclude)
{
    STDMETHOD(Open)(THIS_ D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) PURE;
    STDMETHOD(Close)(THIS_ LPCVOID pData) PURE;
};

DECLARE_INTERFACE is a Windows macro that is defined in ObjBase.h and will expand the previous declaration in C++ like this:

struct ID3DInclude {
 virtual HRESULT __stdcall Open(D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) = 0;

 virtual HRESULT __stdcall Close(LPCVOID pData) = 0;
}; 

Implementing and using this interface in C++ is straightforward:

struct MyIncludeCallback : public ID3DInclude {
 virtual HRESULT __stdcall Open(D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) {
     /// code for Open callback
 }

 virtual HRESULT __stdcall Close(LPCVOID pData) {
     /// code for Close callback
 }
}; 

// Usage
ID3DInclude* include = new MyIncludeCallback();

// Compile a shader and use our Include provider
D3DCompile(..., include, ...);

The hack here is to clearly understand how is layout in memory an instance of ID3DInclude through the Virtual Method Table (VTBL)... Oh, it's really funny to see that the Wikipedia article doesn't use any visual table to represent a virtual table... ok, let's remedy it. If you look at the memory address of an instanciated object, you will find an indirect pointer:

Fig 1. Virtual Method Table layout in memory

So from the pointer to a C++ object implementing pure virtual methods, you will find that the first value is a pointer to a VTBL which is shared among the same type of object (here MyIncludeCallback).

Then in the VTBL, the first value is a pointer to the Open() method implementation in memory. The second to the Close() method.

According to the calling convention, how does look the declaration of this Open() function, if we had to impleement it in pure C?

HRESULT __stdcall MyOpenCallbackFunction(void* thisObject, D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) {
     /// code for Open callback
 }

Simply add a "this object" as the 1st parameter of the callback function (which represents a pointer to the MyIncludeCallback instance in memory) and you have a callback at the function level!

You should understand now how we can easily hack this to provide a C++ interface callback in C#

Translation to the C#/.Net world

The solution is fairly simple. In order to be able to pass a C++ Interface callback implemented in C# to an unmanaged function, we need to replicate how the unmanaged world is going to call the unmanaged functions and how It does expect to have an interface layout in memory.

First, we need to define the ID3DInclude interface in pure C#:

public partial interface Include
{
    /// <summary> 
    /// A user-implemented method for opening and reading the contents of a shader #include file. 
    /// </summary> 
    /// <param name="type">A <see cref="SlimDX2.D3DCompiler.IncludeType"/>-typed value that indicates the location of the #include file. </param>
    /// <param name="fileName">Name of the #include file.</param>
    /// <param name="parentStream">Pointer to the container that includes the #include file.</param>
    /// <param name="stream">Stream that is associated with fileName to be read. This reference remains valid until <see cref="SlimDX2.D3DCompiler.Include.Close"/> is called.</param>
    /// <unmanaged>HRESULT Open([None] D3D_INCLUDE_TYPE IncludeType,[None] const char* pFileName,[None] LPCVOID pParentData,[None] LPCVOID* ppData,[None] UINT* pBytes)</unmanaged>
    //SlimDX2.Result Open(SlimDX2.D3DCompiler.IncludeType includeType, string fileNameRef, IntPtr pParentData, IntPtr dataRef, IntPtr bytesRef);
    void Open(IncludeType type, string fileName, Stream parentStream, out Stream stream);

    /// <summary> 
    /// A user-implemented method for closing a shader #include file. 
    /// </summary> 
    /// <remarks> 
    /// If <see cref="SlimDX2.D3DCompiler.Include.Open"/> was successful, Close is guaranteed to be called before the API using the <see cref="SlimDX2.D3DCompiler.Include"/> interface returns. 
    /// </remarks> 
    /// <param name="stream">This is a reference that was returned by the corresponding <see cref="SlimDX2.D3DCompiler.Include.Open"/> call.</param>
    /// <unmanaged>HRESULT Close([None] LPCVOID pData)</unmanaged>
    void Close(Stream stream);
}

Clearly, this is not exactly what we have in C++... but this is how we would use it... through the usage of Stream. An implementation of this interface would provide a Stream for a particular file to include (most of a time, that could be as simple as stream = new FileStream(fileName)).

This interface is public in the C#/.Net API... but internally we are going to use a wrapper of this interface that is going to create manually the object layout in memory as well as the VTBL. This is done in this simple constructor:

/// <summary>
/// Internal Include Callback
/// </summary>
internal class IncludeCallback
{
    public IntPtr NativePointer;
    private Include _callback;
    private OpenCallBack _openCallBack;
    private CloseCallBack _closeCallback;

    public IncludeCallback(Include callback)
    {
        _callback = callback;
        // Allocate object layout in memory 
        // - 1 pointer to VTBL table
        // - following that the VTBL itself - with 2 function pointers for Open and Close methods
        _nativePointer = Marshal.AllocHGlobal(IntPtr.Size * 3);

        // Write pointer to vtbl
        IntPtr vtblPtr = IntPtr.Add(_NativePointer, IntPtr.Size);
        Marshal.WriteIntPtr(_NativePointer, vtblPtr);
        _openCallBack = new OpenCallBack(Open);
        Marshal.WriteIntPtr(vtblPtr, Marshal.GetFunctionPointerForDelegate(_openCallBack ));
        _closeCallBack = new CloseCallBack(Close);
        Marshal.WriteIntPtr(IntPtr.Add(vtblPtr, IntPtr.Size), Marshal.GetFunctionPointerForDelegate(_closeCallBack));
    }

You can clearly see from the previous code that we are allocating a an unmanaged memory that will hold the object VTBL pointer and the VTBL itself... Because we don't need to make 2 allocation (one for the object's vtbl_ptr/data, one for the vtbl), we are laying out the VTBL just after the object itself, like this:

The declaration of the C# delegates are then straightforward from the C++ declaration:

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
private delegate SlimDX2.Result OpenCallBack(IntPtr thisPtr, SlimDX2.D3DCompiler.IncludeType includeType, IntPtr fileNameRef, IntPtr pParentData, ref IntPtr dataRef, ref int bytesRef);

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
private delegate SlimDX2.Result CloseCallBack(IntPtr thisPtr, IntPtr pData);

You just have to implement the Open and Close method in the wrapper and redirect the calls to the managed Include callback, et voila!

Then after, when calling an unmanaged function that required this callback, you just have to wrap an Include instance with the callback like this:

Include myIncludeInstance = ... new ...;

IncludeCallback callback = new IncludeCallback(callback);

// callback.NativePointer is a pointer to the object/vtbl allocated structure
D3D.Compile(..., callback.NativePointer, ...);

Of course, the IncludeCallback is not visible from the public API but is used internally. From a public interface POV, here is how you would use it:

using System;
using System.IO;
using SlimDX2.D3DCompiler;

namespace TestCallback
{
    class Program
    {
        class MyIncludeCallBack : Include
        {
            public void Open(IncludeType type, string fileName, Stream parentStream, out Stream stream)
            {
                stream = new FileStream(fileName, FileMode.Open);
            }

            public void Close(Stream stream)
            {
                stream.Close();
            }
        }

        static void Main(string[] args)
        {
            var include = new MyIncludeCallBack();
            string value = ShaderBytecode.PreprocessFromFile("test.fx", null, include); 
            Console.WriteLine(value);
        }
    }
}

You can have a look at the complete source code here.