Advanced HLSL using closures and function pointers

Shader languages like HLSL, Cg or GLSL are nowadays driving the most powerful processors in the world, but if you are developing with them, you may have been already a little bit frustrated by one of their expressiveness limitations: the common problem of abstraction and code reuse. In order to overcome this problem, solutions so far were mostly using a glue combination of #define/#include preprocessors directives in order to generate combinations of code, permutation of shaders, so called UberShaders. Recently, this problem has been addressed, for HLSL (new in Direct3D11), by providing the concept of Dynamic Linking, and for GLSL, the concept of SubRoutines, For Direct3D11, the new mechanism has been only available for Shader Model 5.0, meaning that even if this could greatly simplified the problem of abstraction, It is unfortunately only available for Direct3D11 class graphics card, which is of course a huge limitation...

But, here is the good news: While the classic usage of dynamic linking is not really possible from earlier version (like SM4.0 or SM3.0), I have found an interesting hack to bring some kind of closures and functions pointers to HLSL(!). This solution doesn't involve any kind of preprocessing directive and is able to work with SM3.0 and SM4.0, so It might be interesting for folks like me that like to abstract and reuse the code as often as possible! But let's see how It can be achieved...

Managed DirectX with Win8 Metro Style App

Since my last post is quite old, it's time to give more insight about the latest features included in the recent release of SharpDX 2.0 beta. This release is a major step for SharpDX as it includes lots of new APIs but also provides a preview of developing Managed DirectX from a Windows 8 Metro style application. In the meantime, I have been pretty busy as I got a job in a gamedev company located in Japan that is developing a new rendering engine in C#... which is using SharpDX for its windows rendering, and that's extremely exciting to work on a project like this (and of course rewarding for all the investement done in SharpDX!).

While SharpDX is starting to cover almost the whole DirectX API as well as some new Windows Multimedia API (like WIC), I put some effort, just after the Microsoft Build conference, on providing a preview of using SharpDX from a Win8 Metro style application, which is a fantastic opportunity for a C#/.Net developer to use DirectX from both a Metro style application and Desktop application using the same API.

Finally SharpDX is also going to get some new APIs and some interesting features in the following months, especially in the performance domain, to reduce as much as possible the difference of performance between a native C++ DirectX application and a managed C# DirectX application, that will deserve a post on its own. But lets start with what has been achieved in the latest 2.0 version!

Official release of SharpDX 1.0

After three months of intense development, I'm really excited to announce the availability of SharpDX 1.0 , a new platform independent .Net managed DirectX API, directly generated from DirectX SDK headers.

This first version can be considered as stable. The Direct3D10 / Direct3D10.1 API has been entirely tested on a large 3D engine that was using previously SlimDX (thanks patapom!). Migration was quite straightforward, with tiny minor changes to the engine's code.


The key features and benefits of this new API are:

• API is generated from DirectX SDK headers : meaning a complete and reliable API and an easy support for future API.
• Full support for the following DirectX API:
• Direct3D10
• Direct3D10.1
• Direct3D11
• Direct2D1 (including custom rendering, tessellation callbacks)
• DirectWrite (including custom client callbacks)
• D3DCompiler
• DXGI
• DXGI 1.1
• DirectSound
• XAudio2
• XAPO
• An integrated math API directly ported from SlimMath
• Pure managed .NET API, platform independent : assemblies are compiled with AnyCpu target. You can run your code on a x64 or a x86 machine with the same assemblies, without recompiling your project.
• Lightweight individual assemblies : a core assembly - SharpDX - containing common classes and an assembly for each subgroup API (Direct3D10, Direct3D11, DXGI, D3DCompiler...etc.). Assemblies are also lightweight.
• C++/CLI Speed : the framework is using a genuine way to avoid any C++/CLI while still achieving comparable performance.
• API naming convention mostly compatible with SlimDX API.
• Raw DirectX object life management : No overhead of ObjectTable or RCW mechanism, the API is using direct native management with classic COM method "Release".
• Easily mergeable / obfuscatable : If you need to obfuscate SharpDX assemblies, they are easily obfusctable due to the fact the framework is not using any mixed assemblies. You can also merge SharpDX assemblies into a single exe using with tool like ILMerge.

You will also find a growing collection of samples in the Samples Gallery of SharpDX. Most notably with some additional support for Direct2D1 and DirectWrite client callbacks.

Instead of providing a monolithic assembly, SharpDX is providing lightweight individual and interdependent assemblies. All SharpDX assemblies are dependent from the core SharpDX assembly. You just need to add the required assemblies to your project, without embedding the whole DirectX API stack. Here is a chart that explains SharpDX assembly dependencies:

Next versions will provide support for DirectInput, XInput, X3DAudio, XACT3.

About performance

Someone asked me how SharpDX compares to SlimDX in terms of performance. Here is a micro-benchmark on two methods, ID3D10Device1::GetFeatureLevel (alias Device.FeatureLevel) and ID3D10Device::CheckCounterInfo (alias Device.GetCounterCapabilities).

The test consist of 100,000,000 calls on each methods (inside a for, with (10 calls to device.FeatureLevel) * 10,000,000 times) and is repeated 10 times and averaged. Repeated two times.
  

Method	SlimDX	SharpDX	SharpDX vs SlimDX
device.FeatureLevel	3700	3650	1,37%
device.GetCounterCapabilities()	4684	4259	9,98%

For FeatureLevel, the test was sometimes around +/-0.5%.
For GetCounterCapabilities(), the main difference between SharpDX and SlimDX implementation is that SlimDX perform a copy from the native struct to .Net struct while SharpDX is directly passing a pointer to the .Net struct.

This test is of course a micro benchmark and doesn't reflect a real-world usage. Some part of the API could be in favor of SlimDX, but I'm pretty confident that SharpDX is much more consistent in the way structures are passed to the native functions, avoiding as much as possible marshaling structures that doesn't need any custom marshaling (unlike SlimDX that is performing most of a time a marshaling between .Net/Native structure, besides they are binary compatible).

Next?

Finally, I'm going to be able to use this project to make some demos with it! Next target is to develop a XNA like based framework based on SharpDX.Direct3D11.

Stay tuned!

SharpDX, a new managed .Net DirectX API available

If you have followed my previous work on a new .NET API for Direct3D 11,  I proposed SlimDX team this solution for the v2 of their framework, joined their team around one month ago, and I was actively working to widen the coverage of the DirectX API. I have been able to extend the API coverage almost up to the whole API, being able to develop Direct2D samples, as well as XAudio2 and XAPO samples using it. But due to some incompatible directions that the SlimDX team wanted to follow, I have decided to release also my work under a separate project called SharpDX. Now, you may wonder why I'm releasing this new API under a separate project from SlimDX?

Well, I have been working really hard on this from the beginning of September, and I explained why in my previous post about Direct3D 11. I have checked-in lots of code under the v2 branch on SlimDX, while having lots of discussion with the team (mostly Josh which is mostly responsible for v2) on their devel mailing list. The reason I'm leaving SlimDX team is that It was in fact not clear for me that I was not enrolled as part of the decision for the v2 directions, although  I was bringing a whole solution (by "whole", I mean a large proof of concept, not something robust, finished). At some point, Josh told me that Promit, Mike and himself, co-founders of SlimDX, were the technical leaders of this project and they would have the last word on the direction as well as for decisions on the v2 API.

Unfortunately, I was not expecting to work in such terms with them, considering that I had already made 100% of the whole engineering prototype for the next API. From the last few days, we had lots of -small- technical discussions, but for some of them, I clearly didn't agree about the decisions that were taken, whatever the arguments I was trying to give to them. This is a bit of disappointment for me, but well, that's life of open source projects. This is their project and they have other plans for it. So, I have decided to release the project on my own with SharpDX although you will see that the code is also currently exactly the same on the v2 branch of SlimDX (of course, because until yesterday, I was working on the SlimDX v2 branch).

But things are going to change for both projects : SlimDX is taking the robust way (for which I agree) but with some decisions that I don't agree (in terms of implementation and direction). Although, as It may sound weird, SharpDX is not intended to compete with SlimDX v2 : They have clearly a different scope (supporting for example Direct3D 9, which I don't really care in fact), different target and also different view on exposing the API and a large existing community already on SlimDX. So SharpDX is primarily  intended for my own work on demomaking. Nothing more. I'm releasing it, because SlimDX v2 is not going to be available soon, even for an alpha version. On my side, I'm considering that the current state (although far to be as clean as It should be) of the SharpDX API is usable and I'm going to use it on my own, while improving the generator and parser, to make the code safer and more robust.

So, I did lots of work to bring new API into this system, including :

• Direct3D 10
• Direct3D 10.1
• Direct3D 11
• Direct2D 1
• DirectWrite
• DXGI
• DXGI 1.1
• D3DCompiler
• DirectSound
• XAudio2
• XAPO

And I have been working also on some nice samples, for example using Direct2D and Direct3D 10, including the usage of the tessellate Direct2D API, in order to see how well It works compared to the gluTessellation methods that are most commonly used. You will find that the code is extremely simple in SharpDX to do such a thing :

using System;
using System.Drawing;
using SharpDX.Direct2D1;
using SharpDX.Samples;

namespace TessellateApp
{
    /// 

    /// Direct2D1 Tessellate Demo.
    /// 
    public class Program : Direct2D1DemoApp, TessellationSink
    {
        EllipseGeometry Ellipse { get; set; }
        PathGeometry TesselatedGeometry{ get; set; }
        GeometrySink GeometrySink { get; set; }

        protected override void Initialize(DemoConfiguration demoConfiguration)
        {
            base.Initialize(demoConfiguration);

            // Create an ellipse
            Ellipse = new EllipseGeometry(Factory2D,
                                          new Ellipse(new PointF(demoConfiguration.Width/2, demoConfiguration.Height/2), demoConfiguration.Width/2 - 100,
                                                      demoConfiguration.Height/2 - 100));

            // Populate a PathGeometry from Ellipse tessellation 
            TesselatedGeometry = new PathGeometry(Factory2D);
            GeometrySink = TesselatedGeometry.Open();
            // Force RoundLineJoin otherwise the tesselated looks buggy at line joins
            GeometrySink.SetSegmentFlags(PathSegment.ForceRoundLineJoin); 

            // Tesselate the ellipse to our TessellationSink
            Ellipse.Tessellate(1, this);

            // Close the GeometrySink
            GeometrySink.Close();
        }


        protected override void Draw(DemoTime time)
        {
            base.Draw(time);

            // Draw the TextLayout
            RenderTarget2D.DrawGeometry(TesselatedGeometry, SceneColorBrush, 1, null);
        }

        void TessellationSink.AddTriangles(Triangle[] triangles)
        {
            // Add Tessellated triangles to the opened GeometrySink
            foreach (var triangle in triangles)
            {
                GeometrySink.BeginFigure(triangle.Point1, FigureBegin.Filled);
                GeometrySink.AddLine(triangle.Point2);
                GeometrySink.AddLine(triangle.Point3);
                GeometrySink.EndFigure(FigureEnd.Closed);                
            }
        }

        void TessellationSink.Close()
        {            
        }

        [STAThread]
        static void Main(string[] args)
        {
            Program program = new Program();
            program.Run(new DemoConfiguration("SharpDX Direct2D1 Tessellate Demo"));
        }
    }
}

This simple example is producing the following ouput :

which is pretty cool, considering the amount of code (although the Direct3D 10 and D2D initialization part would give a larger code), I found this to be much simpler than the gluTessellation API.

You will find also some other samples, like the XAudio2 ones, generating a synthesized sound with the usage of the reverb, and even some custom XAPO sound processors!

You can grab those samples on SharpDX code repository (there is a SharpDXBinAndSamples.zip with a working solutions with all the samples I have been developing so far, with also MiniTris sample from SlimDX).

Hacking Direct2D to use directly Direct3D 11 instead of Direct3D 10.1 API

Disclaimer about this hack: This hack was nothing more than a proof of concept and I *really* don't have time to dig into any kind of bugs related to it.

[Edit]13 Jan 2011, After Windows Update KB2454826, this hack was not working. I have patched the sample to make it work again. Of course, you shouldn't consider this hack for anykind of production use. Use the standard DXGI shared sync keyed mutex instead. This hack is just for fun![/Edit]


If you know Direct3D 11 and Direct 2D - they were released almost at the same time - you already know that there is a huge drawback to use Direct 2D : It's in fact only working with Direct3D 10.1 API (although It's working with older hardware thanks to the new feature level capability of the API).

From a coding user point of view, this is really disappointing that such a good API doesn't rely on the latest Direct3D API... moreover when you know that the Direct3D 11 API is really close to the Direct3D 10.1 API... In the end, more work are required for a developer that would like to work with Direct3D 11, as It doesn't have any more Text API for example, meaning that in D3D11, you have to do it yourself, which isn't a huge task itself, if you go to the easy precalculated-texture-of-fonts generated by some GDI+ calls or whatever, but still... this is annoying specially when you need to display some information/FPS on the screen and you can't wait to build a nice font-texture-based system...

I'm not completely fair with Direct2D interoperability with Direct3D 11 : there is in fact a well known solution proposed by one guy from DirectX Team that imply the use of DXGI mutex to synchronized a surface shared between D3D10.1 and D3D11. I was expecting this issue to be solved in some DirectX SDK release this year, but It seems that there is no plan to release in the near future an update for Direct2D (see my question in the comments and the anwser...)... WP7 and XNA are probably getting much more attention here...

So last week, I took some time on the Direct2D API and found that It's in fact fairly easy to hack Direct2D and redirect all the D3D10.1 API calls to a real Direct3D 11 instance... and this is a pretty cool news! Here is the story of this little hack...