tag:blogger.com,1999:blog-10766436996835218902024-03-14T05:18:47.692+11:00code4kDirectX and C#/.NETxoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.comBlogger34125tag:blogger.com,1999:blog-1076643699683521890.post-5599052546450502382014-09-14T23:58:00.000+11:002014-09-20T18:18:48.281+11:00Why Windows UI Matters: Part 2This post comes into 2 parts:<br />
<ol>
<li><a href="http://code4k.blogspot.com/2014/09/why-windows-ui-matters-part-1.html">Typography</a></li>
<li>Colors and Tiles (this one)</li>
</ol>
Following my first post about <a href="http://code4k.blogspot.jp/2014/09/why-windows-ui-matters-part-1.html">Typography in Windows 8</a>, I would like in this post to talk more about colors and the tile concept in Windows Phone and Windows 8.x.<br />
<br />
If you create a new account on your Windows 8.x, you will get the following start screen (on a 27")<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDh-Dxm5vkPjjZGQv8jQ-2kXMjul2WFU10kjhBm09lSNBR4BMNL4zgshDvjqDKaN0Vm9ks7Ch1K1rnC3s4tCi2_Ahf6YizjTzy7SVMq-biMiVqFr1ju1V5XZdSNqJPhCERYGK7rWNZmB0/s1600/windows81-start1.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDh-Dxm5vkPjjZGQv8jQ-2kXMjul2WFU10kjhBm09lSNBR4BMNL4zgshDvjqDKaN0Vm9ks7Ch1K1rnC3s4tCi2_Ahf6YizjTzy7SVMq-biMiVqFr1ju1V5XZdSNqJPhCERYGK7rWNZmB0/s1600/windows81-start1.jpg" height="360" width="640" /></a></div>
<br />
First remark, <b>it looks empty</b>. We know the problem for a long time: this was designed for smaller form factors - Windows Phone but unfortunately we got it as-is on desktop.<br />
<br />
Second remark, we also start to get a glimpse of the color problem with the Metro/Tile concept: <b>It looks flashy</b>.<br />
<a name='more'></a><br />
Back in 2012, when Windows 8 UI was revealed, one of the first joke we quickly got on social networks was a comparison with an AOL screenshot from 1996: <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8OPTOAJnIBgmUPxWVVs1Wslt01QXHm2If1czUd-MoGoO6q6_HZZV67YylgBVjnj-LxxTMj0Ko_Z4Q-4Rap5VR9i1BBoGEuwprbMxCv-ikzRgmqfQPbCuFeFhMuxGgJUtwEA9Za-j4QzA/s1600/aol.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8OPTOAJnIBgmUPxWVVs1Wslt01QXHm2If1czUd-MoGoO6q6_HZZV67YylgBVjnj-LxxTMj0Ko_Z4Q-4Rap5VR9i1BBoGEuwprbMxCv-ikzRgmqfQPbCuFeFhMuxGgJUtwEA9Za-j4QzA/s1600/aol.png" height="404" width="640" /></a></div>
<br />
And when you see this kind of comparisons immediately popping up all around, It should ring the bell...<br />
<br />
But let's just zoom in the Windows 8 tiles area:<br />
<br />
<div style="clear: both;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi90QdD3njPXJz3wv3VmBXcNbCIApXPxD6kuNrFqt5qmWtb31CJUaKQzmIDywutTROIZssQGlVwyhbuELbvLoEdFPrm7vsrOeNK6s6U-WtrkJYG1FLkIH3NFVduJSSJX_uEDxPoDlQWpzE/s1600/windows81-start2.jpg" height="601" width="640" /></div>
<br />
<br />
If you are a developer, designer or working on colorization, you will immediately see that some colors have been pushed close to the range #FF0000 or #00FF00 or #0000FF. Visually, <b>colors on this screen are attacking us</b>. <br />
<br />
<h3>
Colors</h3>
<br />
I have put this image into the "<a href="http://mkweb.bcgsc.ca/color_summarizer" target="_blank">Image Color Summarizer</a>" by Martin Krzywinski:
<br />
<br />
<div style="clear: both;">
</div>
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
RGB (Win8)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV4cDdMJD30DXf9Jb7meqWtTmatRkd35wIwAbpAQIklrT_r6eqPOKRsFykT9NMv4tHtsAFOOTN5L6H0zarSz2SYrQvFYs0mpHSK4wleb8R9NPQsWEot_5n8fKpJXdZfCcdPlpjZhQ_mLg/s1600/color_summarizer_ur4xsnv-rgb.jpg" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFMIcweHGkO0dVFOcRKPbbmSbtnxQepmoneBSM9bggSZK41F9cVgXOaVHwXP1e13TtbiAmc9BiW9lvGt4p7nWKnwIOiOgisSrMbIi4YxnMRECjQ1aV-Q_g9jaoya2PLOkoFFo-eo5GPng/s1600/axis-hist-rgb.png" /><br />
<br /></div>
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
Value (Win8)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3_AL7em4Jr_BcyUGewSbOLj48XUkogBoEar4VQfrujAVfztXx5YlvZWJPp8-XANPrY1Z2-GnmarMWxtZ5OuQNMqbqQyow2r22v7ipBUMyu1NWEqze0LlZeS0tuFhaNCZU3oUVpx07bcI/s1600/color_summarizer_ur4xsnv-hsv-v.jpg" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitmr4h-3SV1ecm0zhHUW454RIHCjFQNbkFMWlYdwzf37Wr2ZL4jR4W27hPgTERhvCVzYgEtAp-2tHz76XvV_NCHP_BBooZmruiqXbOGwG_rUu0l1XYjXZqkGCqf3bT9ibUXmEk5HgOWQU/s1600/axis-hist-v.png" /><br />
<br /></div>
<div style="clear: both;">
</div>
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
Saturation (Win8)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVzhPyKyBythKZ5B8KEBR4nAVMP3K1JP3nbeVWLWE5PoElTST8uveeVN8EHCZcXGd0sea0a2t0xtkwSUI7zUzaITKfvEYfKjs9JNieoCVz9RswOrjVOfxpTWrFyo_p28nIpksJ6wZOuDk/s1600/color_summarizer_ur4xsnv-hsv-s.jpg" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg0e2QaVADp7p7TYMZRwDNYNUXUZnJbIMZVl9Srirni6SBYHifE6LaVTV8W1hfh9WlxNY_DHBdxl-jPfMpTc3iGcTUfZSyLrZs3GqXoGH9nyJplJfFEwwJ6iLoinMxSWqo78GOxwSMddM/s1600/axis-hist-s.png" /><br />
<br /></div>
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
Hue (Win8)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmVn-SkQ0nOSM6H7i7l3-bVmPu4nUDKnIeVW1F2wFYAUD5bjPMxzdcSUVpZGIL8LbN1V6JghQP-PCH5hEjJgxZCxD9G_AdufRWaN8Ydo32an8a6VKowMGgaKmkxs0Qop7w8qwdLbIt0k8/s1600/color_summarizer_ur4xsnv-hsv-h.jpg" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmYaVgWOnpKgwP3X6e8a31wcxnqL2KEOZcROQLu6pQLIwPeOnmzliWSrSlBp3KldOqH5y0RuCz60uCn98ANYZg16NdIvmAFr7lEQRB7-j_YFo3cO0whcVeNDrUumK76pfrlM2jqDbx1O0/s1600/axis-hist-h.png" /><br />
<br /></div>
<div style="clear: both;">
</div>
<br />
If we pay more attention to the <b>hue</b> and <b>saturation</b>:<br />
<ul>
<li>For the <b>hue</b>: We are getting several spikes over the whole range. Red, Green, Blue, Pink are popping up. </li>
<li>For the <b>saturation</b>: We are getting mostly high values, meaning that the image is highly saturated.</li>
</ul>
This is the immediate feeling we have when looking at this image. This is confirmed by some naive analysis. <b>On Windows 8, dominant colors are coming from the tiles</b>.<br />
<br />
If we perform this kind of analysis with the front-page of an iPad, we will get completely different results: most notably, we won't get these spikes because it depends on the wallpaper image, so usually, we don't put a #FF0000 image on the background: <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0I5uCzfSnSEyOZaUBV6vpPxZj-sSguAM7ACXKdvk9KJH99U4hL37KRCJJ-srcshx1pZmNSYFsayBC-o78WWFmfjLhB7QnzvFG35DLG7-PHcuKyMlyOgYAQ70vsV85CbNAQAF0IrvJE9E/s1600/ipad.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0I5uCzfSnSEyOZaUBV6vpPxZj-sSguAM7ACXKdvk9KJH99U4hL37KRCJJ-srcshx1pZmNSYFsayBC-o78WWFmfjLhB7QnzvFG35DLG7-PHcuKyMlyOgYAQ70vsV85CbNAQAF0IrvJE9E/s1600/ipad.jpg" height="400" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">iPad "Start" Screen</td></tr>
</tbody></table>
<br />
On the iPad screenshot, we can see that icons are colorful, but they are not attacking us, because <b>on the iPad the dominant color is coming from the background</b>. The image is more desaturated than saturated, and we can effectively confirm that spikes don't spawn everywhere in the hue range:<br />
<br />
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
Saturation (iPad)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQzb76dUIGf0j82acsCDlCT1QLfMlAryCOXV4HfvOoTYuS2LQ_wGBV5t5xG98Zw9WOI6zR8f_lsbOfWbCzEDUYeeL2vCrigppauy3dacZwNya2l9RROdtvt-CkOIRf08RWlD0ipCWmvtI/s1600/ipad-color-hsv-saturation.png" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjg0e2QaVADp7p7TYMZRwDNYNUXUZnJbIMZVl9Srirni6SBYHifE6LaVTV8W1hfh9WlxNY_DHBdxl-jPfMpTc3iGcTUfZSyLrZs3GqXoGH9nyJplJfFEwwJ6iLoinMxSWqo78GOxwSMddM/s1600/axis-hist-s.png" /></div>
<div style="float: left; width: 50%;">
<div class="separator" style="text-align: center;">
Hue (iPad)</div>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4349wg9JwDzXAIsx_gifQXYXBKimaTH0zkoUbniSZY81_c0vu7C55AfETO9T5D8ZZ9tDKfXa_hEo7pqPW1_DZikMk0nfUO3yzgmY2Qc-UwMjk8D0WBGE8YBvHXPC64xKmWt_a8qM0uM8/s1600/ipad-hsv-hue.png" /><br />
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmYaVgWOnpKgwP3X6e8a31wcxnqL2KEOZcROQLu6pQLIwPeOnmzliWSrSlBp3KldOqH5y0RuCz60uCn98ANYZg16NdIvmAFr7lEQRB7-j_YFo3cO0whcVeNDrUumK76pfrlM2jqDbx1O0/s1600/axis-hist-h.png" /></div>
<div style="clear: both;">
</div>
<br />
<br />
Again, I'm not an expert in design and UI interface, but I don't feel that vibrant hue/saturation fits well for a welcome/main desktop screen. I don't believe also that this is the only way to provide a flat design interface. Or is it because of its flatness that they need to over-exaggerate colors to make it less "emotionally flat"?<br />
<br />
The use of different hues in the same image with lots of saturated colors could work for a one-time-visit website/logo... but for our welcome OS screen that we are visiting several times a day...it is nothing less than highly repulsive (hence one reason I have setup to boot directly on desktop and have never been using a single Metro app for 2 years). Checking a bit more about saturated color in design guidelines, I found one article "<a href="http://thevisualcommunicationguy.com/2013/05/23/when-to-use-saturated-colors/" target="_blank">When to Use Saturated Colors?</a>" by Curtis Newbold, and he recommends using saturated colors when:<br />
<ul>
<li><b>You need to attract attention</b>: We don't need our start/welcome screem to attract our attention. No, thanks. We are using it mainly for working.</li>
<li><b>You want to create an exciting atmosphere: </b>don't think this is good either...</li>
<li><b>You want to simplify emotional response</b>: I believe that the emotional response so far for the start screen has not been good</li>
</ul>
But<br />
<blockquote class="tr_bq">
<i>Too many saturated colors next to each other can cause eye fatigue [...]</i></blockquote>
Probably we should insist more that we don't want our welcome/start screen to look like a shiny-marketing-website?<br />
<br />
<h3>
Tiles</h3>
<br />
But, this is not all about colors... <b>Tiles are actually accentuating the problem</b>. Instead of having colorful small icons, we get monochromatic big rectangles that are covering several hundred of pixels on our screen. <b>We can't escape from the space they cover</b>!<br />
<br />
The background is completely hidden. We can't customize this. On our good old desktop (or hey, an iPad "Start" screen looks like our good old Windows Desktop with organized icons no?), we can put a wallpaper, and can have icons on top of it, but icons will not hide the wallpaper. On Windows 8 Start Screen, you really don't have the choice.<br />
<br />
In Windows Phone 8.1, they are trying to get around this problem by allowing to put a background image in the tiles...not sure it is working better. Here, a screen shot of a Windows Phone 8.1, WP8 with a background Android, and finally, a screen shot from an Android device:<br />
<br />
<div style="float: left; width: 33%;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYzBaMHc3WfGdgez4A7prrBDvY9_O2foOW9mop_54paqToI-6Iz3pcHwvOOwZ6YJOHQCjEmw-KQ6kmfxwlTAuFat5zlrKAHdFCGqKDcvNE9CGGkB3Aob7vDJOyS4XKPy7CF1_YItKKUAE/s1600/wp81.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYzBaMHc3WfGdgez4A7prrBDvY9_O2foOW9mop_54paqToI-6Iz3pcHwvOOwZ6YJOHQCjEmw-KQ6kmfxwlTAuFat5zlrKAHdFCGqKDcvNE9CGGkB3Aob7vDJOyS4XKPy7CF1_YItKKUAE/s1600/wp81.jpg" height="320" width="192" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windows Phone 8.1</td></tr>
</tbody></table>
</div>
<div style="float: left; width: 33%;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6pE3dTKIG_WHsRG4isi6RhcTEs7JirC1yUezlalti896_DKAu9hYKrxM2BDLdLQtPjGYnbckGS0F6yVUSYqFAeKfmmBKzJae4kHL5L05nahOSkMJa47zCkVa7GTAYKPzHBM3NXlzZTw4/s1600/wp81-android-background.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6pE3dTKIG_WHsRG4isi6RhcTEs7JirC1yUezlalti896_DKAu9hYKrxM2BDLdLQtPjGYnbckGS0F6yVUSYqFAeKfmmBKzJae4kHL5L05nahOSkMJa47zCkVa7GTAYKPzHBM3NXlzZTw4/s1600/wp81-android-background.jpg" height="320" width="192" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windows Phone 8.1 + Android bg</td></tr>
</tbody></table>
</div>
<div style="float: left; width: 33%;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0EpSYyBEOqENFCAgJXFFP4Sztbfc69WdHjrZ_0LEg8ZX_NqYki-qjMJI8lTPSIU7Mokxs9TKw5D83VsxtUeL9hFJxVoTxGy5xZJp3kgt81gzi8XE62ohk_yn6isj0da5njStIpekcwAE/s1600/Android.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0EpSYyBEOqENFCAgJXFFP4Sztbfc69WdHjrZ_0LEg8ZX_NqYki-qjMJI8lTPSIU7Mokxs9TKw5D83VsxtUeL9hFJxVoTxGy5xZJp3kgt81gzi8XE62ohk_yn6isj0da5njStIpekcwAE/s1600/Android.jpg" height="320" width="180" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Android</td></tr>
</tbody></table>
</div>
<div style="clear: both;">
</div>
Maybe for a phone, the Metro UI concept could make sense for some people. While, at the beginning, I found it attractive and "distinctive", I felt quickly bored by the overall theming. Lots of tiles are using the main background color, so we have lots of applications that are a bit more difficult to distinct with others (only by the monochromatic white icon). We could choose to place icons based on their colors, but I'm usually placing icons based on their placement relative to my fingers. I'm also not particularly convinced by the horizontal/vertical black lines as it looks like less smoother than a clean continuous background.<br />
<br />
I was not supposed to talk about next Windows UI, but let's be clear, when I saw some screenshots at Microsoft Build last year about making the tiles integrated on a new Franksteinesk Start Menu, I was frankly not really happy about the idea :<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img src="http://scr.wfcdn.de/10813/Windows-9-Preview-Build-9834-1410433741-0-0.jpg" height="480" style="margin-left: auto; margin-right: auto;" width="640" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Windows 9 Tech Preview Leak (Source: <a href="http://winfuture.de/news,83577.html" target="_blank">WinFuture.de</a>)</td></tr>
</tbody></table>
<br />
I don't know if you are like me about this, but I would really love our main Windows OS playground going back to some fundamentals:<br />
<ul>
<li>Just get rid of the Metro UI, tiles, colors... as they don't match with the overall UI of all other parts of the OS and applications we are using</li>
<li>Better leverage on the desktop area. May be with virtual desktops, it could make more sense (at some point, it will look like an iPad!... oh boy, who could believe this?)</li>
<li>Add more power to the Taskbar. Just one example: sub-folders/apps group (this is not new, I know!)</li>
</ul>
There are also a bunch of old Windows UI discrepancies floating all around that would deserve a separate post. Hope I will have enough motivation to write it, but I need to go back coding, enough blabla about UI design!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com1tag:blogger.com,1999:blog-1076643699683521890.post-61060636642361451332014-09-13T03:03:00.000+11:002014-09-20T18:02:16.017+11:00Why Windows UI Matters: Part 1This post comes into 2 parts:<br />
<ol>
<li>Typography (this one)</li>
<li><a href="http://code4k.blogspot.com/2014/09/why-windows-ui-matters-part-2.html">Colors and Tiles</a></li>
</ol>
You have probably seen some leaks of <a href="http://winfuture.de/news,83580.html">screenshots of the next Windows</a> and while this could be - or not - the technical preview that is going to be released later this month for enterprise preview, I have been a bit shocked by the poorness of the UI visuals. Lots of commentators are saying that Microsoft is usually tweaking the OS UI few months before releasing to the public, fine, but I hope this technical preview will unveil a better sneak peak of next Windows UI. If it does not, that's really an unfortunate move, because all these images are already generating misunderstanding.<br />
<br />
We are lots around waiting and watching for the next Windows. We are not just waiting for the desktop to come back, we are waiting for something that will make us happy and excited! I'm not a UI designer, I'm just a developer using Windows, but I love good visuals and I'm concerned by the OS UI that accompanies me all along more than 12 hours per day!<br />
<br />
I won't comment more on the next Windows OS, but I would like to take the opportunity to share some of the unpleasant things I'm feeling about for the past years with the Windows Metro UI era.<br />
<br />
<a name='more'></a><h3>
Preamble</h3>
<br />
I remember reading the old document of the Windows Phone design philosophy when it came out (I have found just <a href="http://www.ccoo.cat/dona/documentacio/docs_lleis/dogc_24_04_08.pdf">one around</a>) and the first two major pillars of the Metro philosophy were :<br />
<blockquote class="tr_bq">
<i><b>1) Clean, light, open, and fast: </b>It is visually distinctive, contains ample white space, reduces clutter and elevates typography as a key design element<b><br />2) Content, not chrome: </b>It accentuates focus on the content that the user cares most about, making the product simple and approachable for everyone</i></blockquote>
That's the original seed and... original sin.<br />
<br />
First,
considering that a UI OS is like a transit area, a metro or an airport.
I barely spend a whole day in a metro/airport and I don't think I would
love to. While this design philosophy can be successful for some
applications (like news, because this is one place where the content is
more important than the chrome), it is questionable to apply it
<i>everywhere</i>.<br />
<br />
<h3>
Do you <i class="fa fa-heart" style="color: #ff0033;"></i> typography?</h3>
<br />
Let's take one of my favorite example: The settings of Windows Phone and this apply to some extend to the settings of Windows 8.x. Here are the screenshots of all the settings that are scrollable on a windows developer phone I have: <br />
<br />
<div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOPzUtITeNcxPbvtdbM8m27X8UOdFt2MruELSOBulLNs-8BZmiNrAlkcMuia8nPrK2NzqC2toQqAN-rOosrcqQqPfypvfKE0H32q9oDsCxwhub9rwXVpPPMeUpcXFNGBwCN5n5ZtekYxU/s1600/wp_ss_20140409_0001.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOPzUtITeNcxPbvtdbM8m27X8UOdFt2MruELSOBulLNs-8BZmiNrAlkcMuia8nPrK2NzqC2toQqAN-rOosrcqQqPfypvfKE0H32q9oDsCxwhub9rwXVpPPMeUpcXFNGBwCN5n5ZtekYxU/s1600/wp_ss_20140409_0001.jpg" height="320" width="192" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhv69qKv6VPvOXaoF6-u6EOFm4tZPpVss9cUA4wBpWG9wb3rCnExNJ87AutKLfzMlwwmPYJ0cqmMxvY3UAhxuEhR-lkgy-JE3uuNpLa8bTs080IRuIP-m-V1WQEmMh1P6nePRvGb42ofUM/s1600/wp_ss_20140409_0002.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhv69qKv6VPvOXaoF6-u6EOFm4tZPpVss9cUA4wBpWG9wb3rCnExNJ87AutKLfzMlwwmPYJ0cqmMxvY3UAhxuEhR-lkgy-JE3uuNpLa8bTs080IRuIP-m-V1WQEmMh1P6nePRvGb42ofUM/s1600/wp_ss_20140409_0002.jpg" height="320" width="192" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfwssssEhhaKlC8Y24Jda_R0MqmTFWNo6RRmxT2qXST6-pqUKmA5M1n_CudkrLKFVmFaBLNfCm1WTVPrcvHohoh1U_TZ6-PJOAPGYYrW_wMBMG-fJQr1llQg6CJQPGH8BgSUmA5J01xvc/s1600/wp_ss_20140409_0003.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfwssssEhhaKlC8Y24Jda_R0MqmTFWNo6RRmxT2qXST6-pqUKmA5M1n_CudkrLKFVmFaBLNfCm1WTVPrcvHohoh1U_TZ6-PJOAPGYYrW_wMBMG-fJQr1llQg6CJQPGH8BgSUmA5J01xvc/s1600/wp_ss_20140409_0003.jpg" height="320" width="192" /></a></div>
<div class="clear">
</div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiux34NJTzmk8s0to8ra08AsYNEpCyTakAHb0Kb00Oq-_Ptc6yedZDhObGz9MjeT8QpmoFPcF7qR3HSq2UGmzO03YLVCERHoRyz2ttczNT3YGnj8cym9Kn0jHq83PfDPZt1zf99DcoU0Gc/s1600/wp_ss_20140409_0004.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiux34NJTzmk8s0to8ra08AsYNEpCyTakAHb0Kb00Oq-_Ptc6yedZDhObGz9MjeT8QpmoFPcF7qR3HSq2UGmzO03YLVCERHoRyz2ttczNT3YGnj8cym9Kn0jHq83PfDPZt1zf99DcoU0Gc/s1600/wp_ss_20140409_0004.jpg" height="320" width="192" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIZMbBKKHNqYP2rWT8FI87eIt8pAXZlovLAOvbwQde8R-gZpdc7UDcfbfJBuSuYmOabbnrp-kMbi2igrN6mc0coBnwWwhvZYEWswpNRUXGwdTdpzfXSnfSTH14skSSq6eRiuvOeAPL_Js/s1600/wp_ss_20140409_0005.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIZMbBKKHNqYP2rWT8FI87eIt8pAXZlovLAOvbwQde8R-gZpdc7UDcfbfJBuSuYmOabbnrp-kMbi2igrN6mc0coBnwWwhvZYEWswpNRUXGwdTdpzfXSnfSTH14skSSq6eRiuvOeAPL_Js/s1600/wp_ss_20140409_0005.jpg" height="320" width="192" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoWPx1UJGuTfx_KMVy3VmTGhLG8VPo_3YZWanID9q8ywaszP82PGcc96H4mnuPVZiynTfIaAacAkFItiPJHYYwJU7UqDXFYAzngvf6Ph6u1zTm1fsMcUiqWtDf_K0j3QwomH2WcnA6l2I/s1600/wp_ss_20140409_0006.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoWPx1UJGuTfx_KMVy3VmTGhLG8VPo_3YZWanID9q8ywaszP82PGcc96H4mnuPVZiynTfIaAacAkFItiPJHYYwJU7UqDXFYAzngvf6Ph6u1zTm1fsMcUiqWtDf_K0j3QwomH2WcnA6l2I/s1600/wp_ss_20140409_0006.jpg" height="320" width="192" /></a></div>
<div class="clear">
</div>
<div style="display: block; float: left; text-align: left; width: 50%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNQtOEt25PrB8KdPdhS8TEoESbNulYlH0XCa8-4gq-kjwrBBYJY7UnYcClu61VjE0LjpgVov_W8OoGVkAyX6Df904vk-151DJPn0XFjHiXo46M1UaAuivMMu6kq5o3CMBotvk-2YPrcUE/s1600/wp_ss_20140409_0007.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNQtOEt25PrB8KdPdhS8TEoESbNulYlH0XCa8-4gq-kjwrBBYJY7UnYcClu61VjE0LjpgVov_W8OoGVkAyX6Df904vk-151DJPn0XFjHiXo46M1UaAuivMMu6kq5o3CMBotvk-2YPrcUE/s1600/wp_ss_20140409_0007.jpg" height="320" width="192" /></a></div>
<div style="display: block; float: left; text-align: left; width: 50%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj91dLFsR-CrGFisJ8Ets9zF92AjcC8Z3O04T2FtQgy9HtsPEslz0VjPmiGIZ6_k9Nb0ZnQtEdeAOTDvxKtKTs7cbgCkPxetxxAHpL5JW1ryDEcPL6h3OspxTWYxCZyRX5sFHZw3ibbB4/s1600/wp_ss_20140409_0008.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjj91dLFsR-CrGFisJ8Ets9zF92AjcC8Z3O04T2FtQgy9HtsPEslz0VjPmiGIZ6_k9Nb0ZnQtEdeAOTDvxKtKTs7cbgCkPxetxxAHpL5JW1ryDEcPL6h3OspxTWYxCZyRX5sFHZw3ibbB4/s1600/wp_ss_20140409_0008.jpg" height="320" width="192" /></a></div>
<div class="clear">
</div>
</div>
<br />
It is composed of <b>7 different screens </b>(!), <b>48 individual plain text entries</b> just for the front settings! I have just added the second screen for applications, but some sub-screen-settings are suffering the same syndrome... <br />
<br />
I have found this UI to be <b>one of the most unpleasant settings area I have been using for the past years over the three major mobile OS</b>. Every time I have to use this area, I'm struggling to scroll down, scroll up, blinking my eyes to find - and <i>miss</i> the entry I'm looking for. It was even worse with Windows Phone 8, as it didn't have an accessible notification/simplified control center from the home screen, so everyday, I had to deep dive into these settings to find the flight mode (2nd screen, first line!)<br />
<br />
This is where <b>typography is abused</b>. It has <b>only content</b>, and absolutely <b>no chrome</b>. Where are the: <br />
<ul>
<li>categories?</li>
<li>icons?</li>
<li>colors?</li>
</ul>
<br />
Let's just have a look at the Windows 8.1 version. <br />
<br />
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwRJQNsOp_KdrGaXoYOCMBLJj8wSNN06CnV9pYefA89-GmaxS_zJGjNQ0o2Qez9zf-tLf5JsfQbWg0lJHyq7AfAiJ7om6ZjsViGDQK586blbvamOtd0wNhoUHj_lcjlv8b1s9DNcslmXI/s1600/Windows81Settings1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwRJQNsOp_KdrGaXoYOCMBLJj8wSNN06CnV9pYefA89-GmaxS_zJGjNQ0o2Qez9zf-tLf5JsfQbWg0lJHyq7AfAiJ7om6ZjsViGDQK586blbvamOtd0wNhoUHj_lcjlv8b1s9DNcslmXI/s1600/Windows81Settings1.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd4eKo7CbXdTy6Z2G3WlcUwQmuaQ_cB7WN-myTjIucc52SgccsUqlK1Rhrw-K9STxEoN6SIFZz8aZ3snUFzjepcEs9vz9jhD49w0GXh2CgZX2abB-4kkCwsJyB3JrRw3-2UZC4UxEXKzg/s1600/Windows81Settings2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd4eKo7CbXdTy6Z2G3WlcUwQmuaQ_cB7WN-myTjIucc52SgccsUqlK1Rhrw-K9STxEoN6SIFZz8aZ3snUFzjepcEs9vz9jhD49w0GXh2CgZX2abB-4kkCwsJyB3JrRw3-2UZC4UxEXKzg/s1600/Windows81Settings2.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiUsGaH2Hr1ENkZKzA8CFZ3tY29IDXAI7Dte1EQsKx9D2yoA2x6Hkbl5TgznkigcBcAZFI2GiPrJo6PbRqiPQRd6XlBzzD8fojmXYhelUuht7fWgY2SMSC0iGHQ1bger9kMKWgPRiu1DI/s1600/Windows81Settings3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiiUsGaH2Hr1ENkZKzA8CFZ3tY29IDXAI7Dte1EQsKx9D2yoA2x6Hkbl5TgznkigcBcAZFI2GiPrJo6PbRqiPQRd6XlBzzD8fojmXYhelUuht7fWgY2SMSC0iGHQ1bger9kMKWgPRiu1DI/s1600/Windows81Settings3.png" height="320" width="217" /></a></div>
<div class="clear">
</div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg74flPrTXQns7umQYGW4PMO7pg1MFxlAwBYcDQo8asxtsU2ch81A9N-sRm7xuW6KuAf9594VsrsSrF_2gRGsuBJm8QN3LexeizldZ9OwKCclfQe-R5TQAysda6PiPadfdvehQRhvj10cE/s1600/Windows81Settings4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg74flPrTXQns7umQYGW4PMO7pg1MFxlAwBYcDQo8asxtsU2ch81A9N-sRm7xuW6KuAf9594VsrsSrF_2gRGsuBJm8QN3LexeizldZ9OwKCclfQe-R5TQAysda6PiPadfdvehQRhvj10cE/s1600/Windows81Settings4.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy2N5GT86zjKhd0kWFe5EFCK8cjmc4lXb6J_zgnSP39Cz8fKTETXWpK35abLFGSm5UqrSgLOPLV95REfeIacYB8B3nxQVvJYwP2WdLRBMHsLJem18K7xFDklhIop6gHrFWHvzZzQSdgXU/s1600/Windows81Settings8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy2N5GT86zjKhd0kWFe5EFCK8cjmc4lXb6J_zgnSP39Cz8fKTETXWpK35abLFGSm5UqrSgLOPLV95REfeIacYB8B3nxQVvJYwP2WdLRBMHsLJem18K7xFDklhIop6gHrFWHvzZzQSdgXU/s1600/Windows81Settings8.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijWJCbXohZMBf3jUYts8VYEmyfReKebZ16hgKTpKtHwVH2UAQQ2ZriOgZIWzlex9bx1RGKLXWfdq587Fc0PXe-lYGA83vOnpGkKOH-rsHsY0tZyb93-2sJYKCBV_qYOM5advc3nIO5Cgs/s1600/Windows81Settings9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijWJCbXohZMBf3jUYts8VYEmyfReKebZ16hgKTpKtHwVH2UAQQ2ZriOgZIWzlex9bx1RGKLXWfdq587Fc0PXe-lYGA83vOnpGkKOH-rsHsY0tZyb93-2sJYKCBV_qYOM5advc3nIO5Cgs/s1600/Windows81Settings9.png" height="320" width="217" /></a></div>
<div class="clear">
</div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi08-9sL6dPfb_idVvIg0saxWh-W9AO1we1B7wNhjBVH1R932cCe-j_xKzNvLiUQXigChSvgbcXkHQiNU_NOgvpusLAy_ZKj9MpFVJeF0UcKjmTOyd2rpNKfCD1oa22-xWcFbHu2AfhPsM/s1600/Windows81Settings10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi08-9sL6dPfb_idVvIg0saxWh-W9AO1we1B7wNhjBVH1R932cCe-j_xKzNvLiUQXigChSvgbcXkHQiNU_NOgvpusLAy_ZKj9MpFVJeF0UcKjmTOyd2rpNKfCD1oa22-xWcFbHu2AfhPsM/s1600/Windows81Settings10.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM0AXtlV3dLyHf_6_qSMN9h3YNr_Aw1L6Sx1ccebkxPscGvS7biz8-5EfRS6IHMxnqnsvCCT4xZ7zB1yMH1IRDSSXKCiXZb2fGEtac3_iFLC7uLPeGg0aWjuMRnOYxNEIYUpulM1enlXQ/s1600/Windows81Settings11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM0AXtlV3dLyHf_6_qSMN9h3YNr_Aw1L6Sx1ccebkxPscGvS7biz8-5EfRS6IHMxnqnsvCCT4xZ7zB1yMH1IRDSSXKCiXZb2fGEtac3_iFLC7uLPeGg0aWjuMRnOYxNEIYUpulM1enlXQ/s1600/Windows81Settings11.png" height="320" width="217" /></a></div>
<div style="display: block; float: left; width: 33.33%;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtUfpXNgHpil2Mdi3Y7KB1AXlUGPZH-Ve7cJHqY4AelpDRdoNj6MHvzU4FAu3BrgYhS59NsCXYbddzzyDAA1Ko6biTycXBjPpwRqhoR94MXKHQCYcBo-inPHvHSOOTjs3puW1tgnRiy5I/s1600/Windows81Settings12.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtUfpXNgHpil2Mdi3Y7KB1AXlUGPZH-Ve7cJHqY4AelpDRdoNj6MHvzU4FAu3BrgYhS59NsCXYbddzzyDAA1Ko6biTycXBjPpwRqhoR94MXKHQCYcBo-inPHvHSOOTjs3puW1tgnRiy5I/s1600/Windows81Settings12.png" height="320" width="217" /></a></div>
<div class="clear">
</div>
<br />
<br />
Not surprising whenever I have to change settings on my PC, I'm still going to the
good old control panel. While it is lacking flat designs and refreshed
icons, it is still much easier to access all your settings ( and you
have much more there), than going around the new Windows 8.1 Metro
settings.<br />
<br />
Apart for the categories, we feel cold about colors and icons. <b>Why banning them so hard?</b>
<br />
<br />
Let's just have a look on the settings on my Android device:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq1AmUC43KvO87ep1ifnwv5A8e5UGED_5hxGwAaMIXjmQDf8-5z4OLihG4brIBUVo3KSY80PThDvJZlwzeh-6G6zCslJLLIQZiktjujKjwwQyYne71mfb4Kh9vpXRgapmlpFvFEDj7Qyo/s1600/AndroidSettings.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq1AmUC43KvO87ep1ifnwv5A8e5UGED_5hxGwAaMIXjmQDf8-5z4OLihG4brIBUVo3KSY80PThDvJZlwzeh-6G6zCslJLLIQZiktjujKjwwQyYne71mfb4Kh9vpXRgapmlpFvFEDj7Qyo/s1600/AndroidSettings.png" height="400" width="225" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Settings on an Android</td></tr>
</tbody></table>
<br />
Visually, it is a bit more pleasant, even if I'm not a fan of the toggle buttons (not really a flat design), but overall, it is functionally a <b>lot more usable</b>. My grandpa with its weak sight is much more able to handle these settings than the one from Windows.<br />
<br />
Perceptually, psychologically, without being an expert, I believe this is wrong. Many people, probably starting by myself, are not comfortable with pure text, language, reading...etc. My kids that are not yet able to read would not be able to navigate in these settings (even if they should not have to!).<br />
<br />
Our brain have different ways of deciphering information, based on text, form, colors, spacial placement, sound. Some people can perfectly handle all this text, some can't. Leveraging only on typography, on a single axis of chrome, is making some people confused about this.<br />
<br />
This division of content and chrome is hurting more than it sounds, drying the chrome is drying the content! It is essential that <b>the substance and form come together</b>.<br />
<br />
For the next part, let's talk about <span style="font-size: x-large;"><a href="http://code4k.blogspot.jp/2014/09/why-windows-ui-matters-part-2.html">colors and tiles</a></span>!<br />
<br />
Stay tuned!
xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com0tag:blogger.com,1999:blog-1076643699683521890.post-68849784359595935592014-08-14T02:49:00.000+11:002014-08-14T13:47:37.424+11:00Packages vNext: Power-up our .NET BuildsIn the sequel of my previous post "<a href="http://code4k.blogspot.com/2014/05/managing-multiple-platforms-in-visual.html" target="_blank">Managing multiple platforms in Visual Studio</a>", having done lots of cross-platform development in .NET in the recent years both at work or for <a href="http://sharpdx.org/" target="_blank">SharpDX</a> (with platform specific assemblies, PCL, assemblies using native compiled code... etc) while trying to trick and abuse nuget and msbuild as much as possible, I have realized that in order to provide a smooth integration of "build packages", this require to be more tightly integrated at the core of a build system.<br />
<br />
Unfortunately, we only have today a patchwork of this integration, still quite incomplete and far from what it could be, and this is hurting a lot our development process. We really need something brand new here: we have lots of inputs, usecases, and while it is of course not possible to cover every aspects of all build workflows, It is certainly possible to address most of the common issues we are facing today. Let's try to figure out where this could lead!<br />
<br />
<a name='more'></a><br />
<h3>
What is a platform?</h3>
<br />
Hey, looks like <a href="http://en.wikipedia.org/wiki/Computing_platform" target="_blank">Wikipedia definition</a> is quite good:<br />
<blockquote class="tr_bq">
A <b>computing platform</b> is, in the most general sense, whatever
pre-existing environment a piece of software is designed to run within,
obeying its constraints, and making use of its facilities. Typical
platforms include a <a href="http://en.wikipedia.org/wiki/Computer_architecture" title="Computer architecture">hardware architecture</a>, an <a href="http://en.wikipedia.org/wiki/Operating_system" title="Operating system">operating system</a> (OS), and <a href="http://en.wikipedia.org/wiki/Runtime_library" title="Runtime library">runtime libraries</a>.<sup class="reference" id="cite_ref-1"><a href="http://en.wikipedia.org/wiki/Computing_platform#cite_note-1">[1]</a></sup></blockquote>
<br />
So this could be:<br />
<ul>
<li><b>Targeting different CPU</b>: like x86/x64/ARM...</li>
<li><b>Targeting different OS</b>: Windows Desktop, Windows Phone, Windows Store Apps, Android, iOS, XBoxOne OS, PS4...etc.</li>
<li><b>Targeting other specific HW </b>through an existing API (the <i>runtime libraries</i> of the Wikipedia's definition), like GPU through OpenGL, Direct3D, Metal...etc.</li>
</ul>
<h3>
How do we target a platform in .NET? </h3>
<br />
Here is the short story. Our day life is of course a bit more complex.<br />
<br />
For the CPU part:<br />
<ul>
<li>"<code>Any CPU</code>" is most of the time our time-saver (digression: why oh why "<code>Any CPU</code>" must be defined with a space in the solution and expected to be "<code>AnyCPU</code>" without a space in a xxproj?!)</li>
<li>But when we have to use some external native code (dlls), we have to "DllImport" these functions. Problem is: native code comes with target CPUs, so</li>
<ul>
<li>Either the library we are using is on the OS. For example, Dllimport of Direct3D from a .NET application is transparent, as the OS is handling the x86/x64/ARM switch for us</li>
<li>Or using a custom external native dll:</li>
<ul>
<li><i>Best case</i>: We are lucky at being able to "LoadLibrary"(looking at you Windows RT/Store) to preload the x86/x64 dll, and then let the DllImport use the existing loaded dll</li>
<li><i>Lazy/lame case</i>: Patching the environment PATH variable (not always working)</li>
<li><i>Worst case</i>: We are forced to compile our application against x86/x64/ARM because the target platform doesn't support multiple CPU assemblies in the same package (doh! looking at you Windows RT/Store) or DllImport is not working (doh! Silverlight CLR on Windows Phone 8.0), even if 90% of our code could be AnyCPU and we just want a tiny dll function, we are good to compile/distribute 3 packages. That's our life...needless to say, painful.</li>
</ul>
</ul>
</ul>
<br />
For the OS and runtime libraries part:<br />
<ul>
<li>If we are developing a library and lucky at not using any OS specific APIs (looking at you, <b><code>FileStream</code></b>, no longer portable because of the Windows RT/Store mess!), we can go with Portable Class Libraries (PCL). Of course, if we failed to compile to Any CPU, we are good for the next choice.</li>
<li>If we are developing an application (an exe, a dll activity...etc.) or a non <b>PCL</b>-friendly library, we are good to compile against specific tool-chains (the little <b>msbuild </b>files imported at the end of our xxproj, remember?) and assemblies</li>
</ul>
But wait, that's a little short on the real coding journey here: In order to develop, build and distribute cross-platform libraries/applications, we are often juggling through different processes and constraints:<br />
<ol>
<li>Use external assemblies, libraries, tools</li>
<ol>
<li>Most of the time by having an "external" or "deps" folders in our product repo, storing dlls for a specific version, or being able to recompile these dependencies from the sources from an internal repo. Care must be taken about versioning </li>
<li>Potentially integrating them in our build process (UsingTask, pre-source process, post-exe process, ILMerge...etc.)</li>
<li>Potentially using <b>NuGet</b> to get all-in-one packages</li>
<ul>
<li>If we do so, be ready to accept xxproj to be messed up by nuget, in several places (see next part) and prepare to suffer after a package update with our VCS...etc.</li>
</ul>
</ol>
<li>Develop platform/specifc assemblies that requires <b>platform specific projects</b> (for desktop, for WinRT, for WP8.x, for Android, for iOS...) with potentially some cross-platforms parts (<b>PCL</b>) and sometimes with native code to compile and/or to link to.</li>
<ol>
<li><i>Best case</i>: we can build everything from a single solution (sln), and in some cross-platform cases, using the kind of tricks I described in my previous post.</li>
<li><i>Worst case</i>: we need to handle different solutions for different platforms. Sometimes requiring to develop custom tools to synchronize projects between platforms</li>
<li>Depending on some defines, we could have different builds for the same platform (like debug with logs/release no logs... etc.)</li>
</ol>
<li>Use a build system to compile our solution/projects, most of the time using <b>msbuild</b></li>
<ol>
</ol>
<ul>
<li> Potentially to develop custom msbuild targets and distribute them as part of our product</li>
</ul>
<ol>
</ol>
<li>Distribute our work, potentially using <b>NuGet </b>or some installers</li>
<ul>
<li>Prepare to manage custom PowerShell and msbuild target files in NuGet package if you have anything platform specific (like x86) like in <a href="https://github.com/sharpdx/SharpDX/blob/master/Build/SharpDX.targets">SharpDX.targets</a> used by the nuget package.</li>
</ul>
<li>All our work is version controlled right? So every steps above can lead to some specific cases and annoyances (lock the sln, lock this csproj... hm, no, git era dude, merge conflict or die!)</li>
</ol>
<br />
So, we somewhat end-up with:<br />
<ul>
<li><i>Best-case</i>: We have a single PCL library. Go back home from work, kiss your family.<i> </i></li>
<li><i>Social-case</i>: We are publishing our PCL to nuget<i> </i></li>
<li><i>Worst-case</i>:We have (multiple CPU to support) x (multiple OS/Store Rules) x
(multiple platform specifics APIs) assemblies to </li>
<ul>
<li>develop (hey, <code>#ifdef</code> we still love you, you know)</li>
<li>build (hey, <code>Condition="'$(XXX)'=='true'</code>" is our friend, and oh, don't expect to avoid the msbuild's underground, msbuild is like our grandma, she still needs <i>lots </i>of love)</li>
<li>deploy (hey... hm, ok, I gave up, too many options for a one-liner "hey")</li>
</ul>
</ul>
You may have had going through what is described here, you may have to handle much more worst cases than I can ever imagine, but... can we really improve things here?<br />
<br />
<h3>
Build packages vNext</h3>
<h3>
</h3>
<h3>
</h3>
As a preamble, a little note about NuGet. NuGet has been helping a lot in this area and is a super contribution in the develop/build/deploy chain, but NuGet has still to struggle with legacy builds, sometimes not NuGet fault, in particular:<br />
<ul>
<li>We are still referencing lots of assemblies through the regular "Add Reference..." because they don't have nugets</li>
<li>NuGet is much more intrusive in the xxproj files than a simple "Add Reference": It has to store a relative paths (bad), and if the nuget package have target files, it needs to add some significant code to our xxproj (for example, in <a href="https://github.com/sharpdx/SharpDX-Samples/blob/master/WindowsDesktop/Direct3D11/MiniTri/MiniTri.csproj#L67">SharpDX</a>)</li>
<li>NuGet still needs to add references to our packaged assemblies, so if our package "Dummy" has 50 Dummy.ABC.*.dll assemblies, we will see a lot of them in our "References"</li>
<li>NuGet doesn't have a probing path for looking for installed local assemblies, but needs to store the assembly references paths directly into the xxproj and forcing package storage (that can be configured in a nuget.config but still, no probing path). For example, if we move the project in a directory structure, it doesn't compile any more. </li>
<li>NuGet is not VCS friendly. Updating a version of a package can cause *lots* of updates in our xxproj: prey that nobody else is doing the same thing on the same project on a different branch.</li>
</ul>
Also<br />
<ul>
<li>PCL are good because they are surfacing the API, exposing a lightweight cross-platform core.</li>
<li>We still need to live with platform specific assemblies</li>
</ul>
Note that ASP vNext is easing the definition of dependencies and simple project compilation, but it is failing at providing a fully unified and integrated build system that spans over the different problems when developing cross platforms packages with more complex builds.<br />
<br />
So, we can somewhat improve the process here by unifying the <i>old</i> and <i>new</i> in a <b>Package vNext concept</b>.<br />
<br />
A <b>Package vNext</b> would be pretty much as the NuGet package we have today and would contain:<br />
<ul>
<li>A <b>version </b>number</li>
<li>All <b>meta descriptions </b>found in NuGet (Owners, Project urls...etc) </li>
<li><b>Dependencies to other packages/versions</b> </li>
<li>A <b>set of assemblies</b>, compiled for different platforms (or a single platform if it is really platform specific). </li>
<li>Potentially <b>a set of public properties/flags exposed by the package</b> that could be set by the referencing project, and would allow to configure the way the link against this package (some specific assemblies or not...etc. depending on the platform...etc.)</li>
<li>Potentially <b>PDBs with direct source code </b>included in the package (but unlike NuGet, not stored on a PDB Symbol server)</li>
<li>Potentially <b>documentation </b>that would be automatically accessible from the IDE</li>
<li>Potentially <b>user files</b> to add to the current project</li>
<li>Potentially providing different <b>additional build files </b>(msbuild target files), transparently added to the build (but unlike today, not modifying the host msbuild files)</li>
<li>Potentially an <b>install plugin helper </b>(like powershell, but I would
prefer a .NET interface/plugin system instead of the unfriendly
powershell syntax) </li>
<li>Potentially providing <b>IDE extensions </b>(recognized by some IDE, that could provide specific IDE extension for VS or Xamarin Studio...etc.)</li>
<li><b>Working also for C++ package</b>: providing includes, libs...etc. (and here, C++ would gain a *lot*)</li>
<li>Package could be signed (non-modifiable) </li>
</ul>
<br />
All our xxproj project (C#, VB, F#...etc.) would <b>reference a package vNext</b> (but usually not a path to package, though it could be possible in some cases), just like this:<br />
<br />
<pre class="brush: xml"><ItemGroup>
<!-- Package loaded from probing paths -->
<PackageReference Include=".NET" Version="4.0" />
<PackageReference Include="YourPackage" Version="1.0" />
<!-- Package loaded from probing paths but with the version defined at solution level -->
<PackageReference Include="YourPackageSpecialVersion" />
<!-- Package loaded from specific path -->
<PackageReference File="path/to/location/FixedLocalPackage-1.0.0" />
</ItemGroup>
...
<Import Project="$(MSBuildToolsPath)\Microsoft.CommonvNext.targets" />
</pre>
<br />
This is the <b>only modification that would be required to reference a package</b>. <b>Everything else</b> (target, custom tasks, files...etc.) would be <b>automatically handled and integrated by the build system</b> (here the CommonvNext.targets).<br />
<br />
When we are targeting a platform specific application, or providing a PCL library, this should be only specified by some properties at the beginning of the project. We would not have to reference explicit targets/dlls in the project (currently, we need to have include CSharp.targets, or Xaml.targets, or WindowsPhone.targets...etc.) but handled by the build system.<br />
<br />
The <b>package version could be defined directly at:</b><br />
<ul>
<li><b>the xxproj project level</b> </li>
<li><b>the solution level</b>, in order to avoid the multiplication of versions all around in all projects from a solution (like a sealed version that could not be override unless specified explicitely with an "override" attributes, exactly like in our languages) </li>
</ul>
A <b>package local probing path</b> would be used in the same way the PATH is used to locate native dlls. This probing path could be:<br />
<ul>
<li>Provided by the system</li>
<li>Override locally at the solution level</li>
<li>Override locally at the project level</li>
</ul>
Like <b>NuGet</b>, It would be possible to query <b>a remote probing path</b> in order to automatically download missing packages.<br />
<ul>
</ul>
<b>Package vNext in the probing path would not be expanded/unzip to the disk</b>, at least visible to the user. Instead they would stay just single plain files (unlike NuGet that is requiring to explicitly expand the packages in a "visible" folder). It is the build infrastructure that would take care to transparently unzip them in some places (for example, in a .vstmp folder at the root of a solution, easily ignorable from a VCS, or on a fixed central temp repo on the disk... etc.)<br />
<br />
When <b>compiling a project to target multiple platforms</b>, the IDE should provide a way to easily identify which files is going to which platform from a xxxproj. This is a bit orthogonal to the Package vNext, but quite important to it if we want a project to target multiple platforms easily.<br />
<b>Packaging and publishing a Package vNext</b> should be part of the build system, as for NuGet that is using nuspec files or directly xxproj files. It means that building a solution, or a project, would produce one or several <b>package vNext</b> directly consumable by other projects. A Package itself in a solution could contain one or several projects...etc. But a project would reference other packages, not projects.<br />
<br />
<br />
<b>Digression on implem of such a system with current msbuild system</b>: One limitation of msbuild is that it cannot import a variable list of *.targets files, all this list must be known at compile time. But, a workaround would be that the build system would generate an intermediate build files (only used internally), exactly like it does for solution files (that are converted into a single msbuild files when building a solution).<br />
<br />
With such a system, we would be able:<br />
<ul>
<li>To develop a cross platform application from a single solution, and even from a same project able to target multiple platforms</li>
<li>To enhance the experience of working with libraries (core .NET framework, external libs...etc.) a unified system instead of having several systems/workarounds (add reference, target files, nuget packages)</li>
<li>To reduce the changes/friction on xxproj, when we are switching package versions...etc. leading to much more VCS friendly build system</li>
</ul>
<h3>
A build dream to build!</h3>
<h3>
</h3>
<h3>
</h3>
Ok, let's face it: This post is describing a "nice to have" concept. It is always easy to write this <i>scratching</i> article, but way more difficult to implement it! When looking at NuGet source code, we can see that it is *lots* of work to provide this kind of infrastructure. <br />
<br />
Still, I believe that a full integration of the notion of package is a key direction for developing, building and deploying cross platform/platform-specific applications in .NET and we should embrace it at the core of our build system.<br />
<br />
So what do you think about this? I'm sure there are lots of ideas that could come to improve all this concept, please, share it!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com0tag:blogger.com,1999:blog-1076643699683521890.post-25421273539317786412014-06-16T01:21:00.000+11:002014-07-01T12:51:20.307+11:00Micro-benchmarking .NET Native and RyuJit<i><span style="font-size: small;">[Disclaimer] As both RyuJIT and .NET Native are improving a lot between their updates/previews, results of benchmarks in this post may no longer be relevant. Benchmarks were ran with the following versions:</span></i><br />
<ul><i><span style="font-size: small;">
</span></i>
<li><span style="color: #cc0000; font-size: small;">RyuJIT CTP4 </span></li>
<span style="font-size: small;">
<span style="color: #cc0000;">
</span>
</span>
<li><span style="color: #cc0000; font-size: small;">.NET Native developer preview 2</span><span style="font-size: small;"> </span></li>
</ul>
<br />
While .NET JIT is performing quite well on Windows, it is still behind a fully optimized C++ program, though an efficient compiled program is not only about code generation but also memory management and data locality, the .NET Team has recently introduced two new technologies that might help for the code gen part: The introduction of <a href="http://blogs.msdn.com/b/dotnet/archive/tags/dotnetnative/" target="_blank">.NET Native</a>, an offline .NET compiler (similar to ngen, but using the backend optimizer that is used by the C++ compiler) and the next generation of .NET JIT called "<a href="http://blogs.msdn.com/b/dotnet/archive/tags/ryujit/" target="_blank">RyuJit</a>". In this post I would like to present the results of some micro-benchmarks that are roughly evaluating some performance benefits of these two new technologies.<br />
<br />
First of all, you may have already read about a few benchmarks results available around about RyuJit and .NET Native, here is a non-exhaustive list I have found (if you have more pointers, let me know!):<br />
<ul>
<li><a href="http://frankniemeyer.blogspot.com/2014/04/a-first-look-at-ruyjit-ctp3-and-simd.html" target="_blank">A first look at RyuJIT CTP3 and SIMD (SSE2) support for .NET</a> by Frank Niemeyer</li>
<li><a href="http://blogs.msdn.com/b/clrcodegeneration/archive/2014/02/26/lies-damn-lies-and-benchmarks.aspx" target="_blank">Lies, damn lies and benchmarks</a> by Kevin Frei </li>
<li><a href="http://java.dzone.com/articles/net-native-performance-and" target="_blank">.NET Native Performance</a> by Sasha Goldstein</li>
</ul>
<ul>
</ul>
<br />
<a name='more'></a><h3>
The micro-benchmark protocol</h3>
<br />
Micro-benchmarking is not the best way to give a measure of the overall benefits, but it can help to dig into some particular patterns. For this benchmark, I haven't developed a new one but instead built a "freak-benchmark" composed of some micro-benchmarks I found on Internet, mainly:<br />
<ul>
<li>"<a href="http://www.codeproject.com/Articles/212856/Head-to-head-benchmark-Csharp-vs-NET" target="_blank">Head-to-head benchmark: C++ vs .NET</a>" by Qwertie has a nice collection of micro-benchmarks. So I decided to use it as a basis</li>
<li>"<a href="http://www.cs.ucsb.edu/~ckrintz/racelab/PhxCSBenchmarks/" target="_blank">A Collection of Phoenix-Compatible C# Benchmarks</a>" I used the port of a subset of Java Grande benchmarks. </li>
<li>Two custom benchmarks measuring the cost of interop which is important in cases where you can possibly call lots of native methods (which is the case when using SharpDX for example)</li>
</ul>
RyuJit is also coming with SIMD, but I will reserve a dedicated post to test this new feature.<br />
<br />
I don't claim that these micro-benchmarks are exhaustive nor they are all correctly implemented (some of the JavaGrande benchmark seems to be not robust), but as we are measuring relative performance, that should be fine. In the end we just want to know how much .NET Native or RyuJit can perform compare to the same program running on the legacy JIT.<br />
<br />
Also as both .NET Native and RyuJit are in development, we can't really draw any definitive conclusions.<br />
<br />
.NET Native is only available for Windows Store App while RyuJit is only available on x64, hence the platforms tested in this bench are:<br />
<ul>
<li>.NET 32 Desktop</li>
<li>.NET 32 AppStore</li>
<li>.NET 32 AppStore Native</li>
<li>.NET 64 Desktop</li>
<li>.NET 64 AppStore</li>
<li>.NET 64 AppStore Native</li>
<li>.NET 64 Desktop RyuJit </li>
</ul>
<br />
.NET32/.NET64 using .NET Framework 4.5.1. The machine is an Intel(R) Core(TM) i7-4770 CPU @ 3.4GHz with 16Go of RAM.<br />
<br />
The source of these benchmarks is available on GitHub <a href="https://github.com/xoofx/BenchNativeApp" target="_blank">BenchNativeApp</a>. <br />
<h3>
Comparison .NET32 (x86)</h3>
<br />
Comparison between:<br />
<ul>
<li>.NET 32 Desktop</li>
<li>.NET 32 AppStore</li>
<li>.NET 32 AppStore Native</li>
</ul>
Normalized with performance relative to desktop. Higher is better. (2.0 means that a test is 2 times faster than desktop). I checked the standard dev was in some reasonable range.<br />
<ul>
<li>In <span style="color: #6aa84f;">green</span>, results above +10%</li>
<li>In <span style="color: #e06666;">red</span>, results below -10% </li>
</ul>
<br />
<table border="1" cellpadding="0" cellspacing="0" style="width: 522px;"><colgroup><col style="mso-width-alt: 10349; mso-width-source: userset; width: 212pt;" width="283"></col>
<col style="mso-width-alt: 2486; mso-width-source: userset; width: 51pt;" width="68"></col>
<col style="mso-width-alt: 2742; mso-width-source: userset; width: 56pt;" width="75"></col>
<col style="mso-width-alt: 3510; mso-width-source: userset; width: 72pt;" width="96"></col>
</colgroup><tbody>
<tr height="40" style="height: 30.0pt;">
<td class="xl65" height="40" style="height: 30.0pt; width: 212pt;" width="283">Name</td>
<td class="xl67" style="width: 51pt;" width="68"><div style="text-align: center;">
.NET 32 </div>
(Desktop)</td>
<td class="xl67" style="text-align: center; width: 56pt;" width="75">.NET 32 <br />
(AppStore)</td>
<td class="xl67" style="text-align: center; width: 72pt;" width="96">.NET 32 Native <br />
(AppStore)</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.90</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.72</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.81</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
<td class="xl66">0.92</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 0 Ints to
strings</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.79</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.13</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 1
Adding/setting</td>
<td class="xl66">1.00</td>
<td class="xl66">0.91</td>
<td class="xl66">1.02</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 2 Running
queries</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.83</td>
<td class="xl66">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 3 Removing
items</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.85</td>
<td class="xl66">1.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.89</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.90</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.77</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
<td class="xl66">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: uint<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.03</td>
<td class="xl66">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: ulong<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.94</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.51</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: float<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.86</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.23</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.55</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.04<span style="background-color: #38761d;"></span></td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: long<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.83</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">3.34</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.30</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.93</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int via IMath<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.57</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int without generics<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 3 Parse
(x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 4 Sort (x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.09</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Interface
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.71</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: No-inline
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.03</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Static NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;"><span style="mso-spacerun: yes;"> </span>no-op</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Virtual
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.11</td>
<td class="xl66">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<double>[n][n]<span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.43</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<float>[n][n]<span style="mso-spacerun: yes;"> </span></float></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.68</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<int>[n][n]<span style="mso-spacerun: yes;"> </span></int></td>
<td class="xl66">1.00</td>
<td class="xl66">0.95</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.59</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
Array2D<double><span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.64</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n*n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.78</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">1.04</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: int[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.21</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">09-Sudoku<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.04</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">10-Polynomials<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.03</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">11-JGFArithBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">23.81</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">12-JGFAssignBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.80</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.20</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">13-JGFCastBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.27</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">14-JGFCreateBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.98</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.81</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">15-JGFFFTBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">16-JGFHeapSortBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66">1.01</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">17-JGFLoopBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.04</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">18-JGFRayTracerBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.98</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.88</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">19-float4x4 matrix mul, Managed Standard</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.63</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">20-float4x4 matrix mul, Managed
unsafe<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.96</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">21-float4x4 matrix mul, Interop Standard</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.23</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.42</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">22-float4x4 matrix mul, Interop SSE2<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.36</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.82</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">23-managed add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">7.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">24-managed no-inline add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">25-interop add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.21</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">26-interop indirect add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.26</td>
</tr>
</tbody></table>
<br />
<h4>
Quick analysis</h4>
We would probably expect a column full of green lights for the .NET Native, but this is unfortunately not the case! Some notes:<br />
<ul>
<li>.NET Native is as efficient as a C++ compiler at coalescing arithmetic instructions (test 11, or 23). Basically the test 23 is able to reduce the addition set of <code>x+=1, x+=2, x+=-3, x+=1, x+=2, x+=-3, x+=1</code> to a single <code>x+= 1</code>, resulting in some impressive speedup. Coalescing of instructions is probably the factor that is helping in most tests there.</li>
<li>Some float/double x87 calculations seems to perform badly with .NET Native. </li>
<li>Pure interop seems slightly more efficient, which is good whenever we are frequently calling native functions (like when using SharpDX/Direct3D11). Note that indirect interop (wrapping a DllImport by another function) is also faster which is great, as It was an issue with current interop that were not inlined by the JIT when they are wrapped, resulting in lots of duplicate prologue/epilogue code for unmanaged/managed transitions (while when it is correctly inlined, consecutive access to interop functions are handled in group when switching context unmanaged/managed)</li>
<li>Some tests are 2x times slower with .NET Native, though I haven't look at the generated x86 code.</li>
</ul>
Overall it is still promising, we can see some significant boosts in some tests while some others are performing a bit worse.<br />
<br />
<h3>
Comparison .NET64 (x64)</h3>
<br />
Comparison between:<br />
<ul>
<li>.NET 64 Desktop</li>
<li>.NET 64 AppStore</li>
<li>.NET 64 AppStore Native</li>
<li>.NET 64 Desktop RyuJit </li>
</ul>
Normalized with performance relative to desktop. Higher is better. (2.0 means that a test is 2 times faster than desktop)<br />
<ul>
<li>In <span style="color: #6aa84f;">green</span>, results above +10%</li>
<li>In <span style="color: #e06666;">red</span>, results below -10% </li>
</ul>
<br />
<table border="1" cellpadding="0" cellspacing="0" style="width: 610px;"><colgroup><col style="mso-width-alt: 9984; mso-width-source: userset; width: 205pt;" width="273"></col>
<col style="mso-width-alt: 2596; mso-width-source: userset; width: 53pt;" width="71"></col>
<col style="mso-width-alt: 2742; mso-width-source: userset; width: 56pt;" width="75"></col>
<col style="mso-width-alt: 3620; mso-width-source: userset; width: 74pt;" width="99"></col>
<col style="mso-width-alt: 3364; mso-width-source: userset; width: 69pt;" width="92"></col>
</colgroup><tbody>
<tr height="40" style="height: 30.0pt;">
<td class="xl65" height="40" style="height: 30.0pt; width: 205pt;" width="273">Name</td>
<td class="xl67" style="width: 53pt;" width="71">.NET 64<br />
<span style="mso-spacerun: yes;"> </span>(Desktop)</td>
<td class="xl67" style="width: 56pt;" width="75">.NET 64<br />
(AppStore)</td>
<td class="xl67" style="width: 74pt;" width="99">.NET 64 Native<br />
(AppStore)</td>
<td class="xl67" style="width: 69pt;" width="92">.NET 64 RyuJit<br />
(Desktop)</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.04</td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66">0.91</td>
<td class="xl66">1.00</td>
<td class="xl66">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.95</td>
<td class="xl66">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 0 Ints to
strings</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.69</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.72</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 1
Adding/setting</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.85</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.84</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 2 Running
queries</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.82</td>
<td class="xl66">0.90</td>
<td class="xl66">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 3 Removing
items</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.81</td>
<td class="xl66">0.91</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.98</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.10</td>
<td class="xl66">1.04</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
<td class="xl66">1.06</td>
<td class="xl66">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">1.02</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.15</td>
<td class="xl66">1.03</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: uint<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.97</td>
<td class="xl66">0.94</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: ulong<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.15</td>
<td class="xl66">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4.20</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: float<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.36</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.91</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.42</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.21</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">5.19</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.83</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.89</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: long<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
<td class="xl66">0.93</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.34</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.33</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.29</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.98</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.28</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int via IMath<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.65</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int without generics<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.70</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 3 Parse
(x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.50</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 4 Sort (x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.95</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.30</td>
<td class="xl66">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Interface
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.69</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.85</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: No-inline
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.92</td>
<td class="xl66">0.96</td>
<td class="xl66">0.96</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Static NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">Not Applicable</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.20</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Virtual
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.92</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.74</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<double>[n][n]<span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.14</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.17</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<float>[n][n]<span style="mso-spacerun: yes;"> </span></float></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">5.01</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<int>[n][n]<span style="mso-spacerun: yes;"> </span></int></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.34</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
Array2D<double><span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">3.83</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.75</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n*n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66">0.96</td>
<td class="xl66">0.98</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: int[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.19</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">09-Sudoku<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.38</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.48</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">10-Polynomials<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.94</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">11-JGFArithBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">12-JGFAssignBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.53</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">13-JGFCastBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.39</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">14-JGFCreateBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.81</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">15-JGFFFTBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.18</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">16-JGFHeapSortBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">17-JGFLoopBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.08</td>
<td class="xl66">1.01</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">18-JGFRayTracerBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.87</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.13</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">19-float4x4 matrix mul, Managed Standard</td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66">1.04</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.36</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">20-float4x4 matrix mul, Managed
unsafe<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
<td class="xl66">0.90</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">21-float4x4 matrix mul, Interop Standard</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.20</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.46</td>
<td class="xl66">1.03</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">22-float4x4 matrix mul, Interop SSE2<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.36</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.92</td>
<td class="xl66">1.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">23-managed add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">24-managed no-inline add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.89</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.48</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">25-interop add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.17</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.11</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">26-interop indirect add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.28</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.38</td>
</tr>
</tbody></table>
<br />
<h4>
Quick analysis</h4>
Slightly better than x86 code gen, the .NET Native x64 and RyuJit are on average performing better than their JIT counterpart. Some notes:<br />
<ul>
<li>Unexpectedly, coalescing
of arithmetic instructions (test 11, or 23) is not happening for .NET Native, but for RyuJit.</li>
<li>Performance on float/double is better. Most likely SSE registers are better used.</li>
<li>Sudoku tests is getting a nice +40-50% faster with .NET Native and RyuJit</li>
</ul>
<br />
<br />
<h3>
Comparison .NET32 Native vs .NET64 Native </h3>
Just use .NET 32 Native as a reference (1.0) and compare it to the .NET 64 Native.<br />
Normalized with performance relative to .NET 32 Native. Higher is better. (2.0 means that a test on x64 Native is 2 times faster than x86 )<br />
<ul>
<li>In <span style="color: #6aa84f;">green</span>, results above +10%</li>
<li>In <span style="color: #e06666;">red</span>, results below -10% </li>
</ul>
<br />
<table border="1" cellpadding="0" cellspacing="0" style="width: 523px;"><colgroup><col style="mso-width-alt: 10349; mso-width-source: userset; width: 212pt;" width="283"></col>
<col style="mso-width-alt: 4388; mso-width-source: userset; width: 90pt;" width="120"></col>
<col style="mso-width-alt: 4388; mso-width-source: userset; width: 90pt;" width="120"></col>
</colgroup><tbody>
<tr height="40" style="height: 30.0pt;">
<td class="xl68" height="40" style="height: 30.0pt; width: 212pt;" width="283">Name</td>
<td class="xl67" style="width: 90pt;" width="120">.NET 32 <br />
Native (AppStore)</td>
<td class="xl67" style="width: 90pt;" width="120">.NET 32 vs 64<br />
Native (AppStore)</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td class="xl65" height="20" style="height: 15.0pt;">00-Big int Dictionary: 1
Adding items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.31</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.35</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.19</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 0 Ints to
strings</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.13</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 1
Adding/setting</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.22</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 2 Running
queries</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.17</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 3 Removing
items</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 2 Running queries</td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.96</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: uint<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: ulong<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: float<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.40</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.49</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: long<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">7.65</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.01</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int via IMath<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int without generics<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 3 Parse
(x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 4 Sort (x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.96</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Interface
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: No-inline
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.20</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Static NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">not applicable</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Virtual
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<double>[n][n]<span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.29</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<float>[n][n]<span style="mso-spacerun: yes;"> </span></float></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.24</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<int>[n][n]<span style="mso-spacerun: yes;"> </span></int></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.38</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
Array2D<double><span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n*n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.30</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: int[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.37</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">09-Sudoku<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.13</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">10-Polynomials<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">11-JGFArithBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.61</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">12-JGFAssignBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.86</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">13-JGFCastBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.57</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">14-JGFCreateBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.93</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">15-JGFFFTBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">16-JGFHeapSortBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.08</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">17-JGFLoopBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">18-JGFRayTracerBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">19-float4x4 matrix mul, Managed Standard</td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.34</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">20-float4x4 matrix mul, Managed
unsafe<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66">1.02</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">21-float4x4 matrix mul, Interop Standard</td>
<td class="xl66">1.00</td>
<td class="xl66">1.09</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">22-float4x4 matrix mul, Interop SSE2<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.15</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">23-managed add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.16</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">24-managed no-inline add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.35</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">25-interop add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.15</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">26-interop indirect add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl66">1.00</td>
<td class="xl66" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.15</td>
</tr>
</tbody></table>
<br />
<h4>
Quick analysis</h4>
.NET 64 Native code gen is better than .NET 32 Native code gen. Haven't dig into code gen, but more registers for x64 might help optim while x86 is still fighting with a limited set of registers (and x86 code is not using SSE instructions, so it doesn't help). Good to see that interop is also better on x64, while it is not the case for JIT x64 where it is usually much slower.<br />
<br />
<h3>
Comparison .NET64 Native vs .NET64 RyuJit</h3>
<br />
Use .NET 64 Native as a reference (1.0) and compare it to the .NET 64 RyuJit.<br />
Normalized
with performance relative to .NET 64 Native. Higher is better. (2.0
means that a test on x64 RyuJit is 2 times faster than x64 Native )<br />
<ul>
<li>In <span style="color: #6aa84f;">green</span>, results above +10%</li>
<li>In <span style="color: #e06666;">red</span>, results below -10% </li>
</ul>
<br />
<table border="1" cellpadding="0" cellspacing="0" style="width: 443px;"><colgroup><col style="mso-width-alt: 9984; mso-width-source: userset; width: 205pt;" width="273"></col>
<col style="mso-width-alt: 2742; mso-width-source: userset; width: 56pt;" width="75"></col>
<col style="mso-width-alt: 3474; mso-width-source: userset; width: 71pt;" width="95"></col>
</colgroup><tbody>
<tr height="60" style="height: 45.0pt;">
<td class="xl66" height="60" style="height: 45.0pt; width: 205pt;" width="273">Name</td>
<td class="xl65" style="width: 56pt;" width="75">.NET 64 <br />
Native (AppStore)</td>
<td class="xl65" style="width: 71pt;" width="95">.NET 64 vs 64<br />
RyuJit<span style="mso-spacerun: yes;"> </span></td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td class="xl63" height="20" style="height: 15.0pt;">00-Big int Dictionary: 1
Adding items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.02</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 2 Running queries</td>
<td class="xl64">1.00</td>
<td class="xl64">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">00-Big int Dictionary: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 0 Ints to
strings</td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.38</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 1
Adding/setting</td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.18</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 2 Running
queries</td>
<td class="xl64">1.00</td>
<td class="xl64">1.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">01-Big string Dictionary: 3 Removing
items</td>
<td class="xl64">1.00</td>
<td class="xl64">1.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 1 Adding
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.94</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 2 Running queries</td>
<td class="xl64">1.00</td>
<td class="xl64">0.91</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">02-Big int sorted map: 3 Removing
items<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.14</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.89</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: uint<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.97</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">03-Square root: ulong<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.83</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.26</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: float<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.73</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.56</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: FPL16<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4.31</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">04-Simple arithmetic: long<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.96</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: double<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: FPI8<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.78</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.78</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int via IMath<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.53</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">05-Generic sum: int without generics<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.59</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 3 Parse
(x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">06-Simple parsing: 4 Sort (x1000000)<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.73</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Interface
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.24</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: No-inline
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Static NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.00</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">07-Trivial method calls: Virtual
NoOp<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.81</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<double>[n][n]<span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl64">1.00</td>
<td class="xl64">1.03</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<float>[n][n]<span style="mso-spacerun: yes;"> </span></float></td>
<td class="xl64">1.00</td>
<td class="xl64">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
<int>[n][n]<span style="mso-spacerun: yes;"> </span></int></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.86</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply:
Array2D<double><span style="mso-spacerun: yes;"> </span></double></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.72</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n*n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.99</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: double[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.02</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">08-Matrix multiply: int[n][n]<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.94</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">09-Sudoku<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.07</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">10-Polynomials<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">11-JGFArithBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">1.10</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">12-JGFAssignBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.52</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">13-JGFCastBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.40</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">14-JGFCreateBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.23</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">15-JGFFFTBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.98</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">16-JGFHeapSortBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.98</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">17-JGFLoopBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.93</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">18-JGFRayTracerBench<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.30</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">19-float4x4 matrix mul, Managed Standard</td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.31</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">20-float4x4 matrix mul, Managed
unsafe<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1.12</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">21-float4x4 matrix mul, Interop Standard</td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.71</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">22-float4x4 matrix mul, Interop SSE2<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.55</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">23-managed add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #C6EFCE; color: #006100; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4.05</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">24-managed no-inline add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.68</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">25-interop add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64">0.95</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt;">26-interop indirect add<span style="mso-spacerun: yes;"> </span></td>
<td class="xl64">1.00</td>
<td class="xl64" style="background: #FFC7CE; color: #9c0006; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">0.30</td>
</tr>
</tbody></table>
<br />
<h4>
Quick analysis</h4>
<br />
Surprisingly, RyuJit is performing quite well or sometimes even better than .NET 64 Native. Might be interesting to dig into this. <br />
<br />
<h3>
Summary</h3>
<br />
As both .NET Native and RyuJit are still in alpha/beta stages, we can't really assert any definitive conclusions here. We can see a trend of improvements in some specific areas, while some tests are still performing a bit worse than the legacy JIT. <i><span style="color: #660000;">[Edit]The release of .NET Native Developer Preview 3 on June 30 2014, is showing some improvements in code gen, so .NET Native and RyuJIT are definitely being improved between updates and it is great! [/Edit] </span></i><br />
<br />
It is good to see.NET 64 getting better and performing well with .NET Native and RyuJit, Until now I have been a bit reluctant at using it, but it looks more robust compare to x86 code gen.<br />
<br />
While code gen can be undoubtedly improved with offline compilers or a more modern JIT like RyuJit, we probably can't expect the moon. As I said in this introduction, code gen is only a part of the overall performance cake. The other part, that is most likely not yet covered by these new compiler architectures, is <i>data locality</i>: things like ability to create fat objects - embed instantiation of objects instance into another instance - or creation of short live objects (not value types) on the stack instead of the heap are still areas where .NET can probably be improved. I will hopefully take more time in a next post to explain why this is an important area of improvements and what could be done.<br />
<br />
Anyway, this is great to see .NET performance back into the ring! I'm eager also to be able to use .NET Native on desktop.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com4tag:blogger.com,1999:blog-1076643699683521890.post-84481691954431824822014-05-25T20:08:00.000+11:002014-08-14T02:52:35.226+11:00Managing multiple platforms in Visual StudioWho has not struggled to correctly manage multiple platform configurations in Visual Studio without ending to edit a solution file or tweak some msbuild files by hand? Recently, I decided to cleanup the antique <code>SharpDX.sln</code> in SharpDX that was starting to be a bit fat and not easy to manage. The build is not extremely bizarre there, but as it needs to cover the combinations of NetPlatform x OSPlatform x DirectXVersion x Debug/Release with around 40 projects (without the samples), it is an interesting case of study. It turns out that modifying the solution to <b>make a clean multi-platform build was impossible without hacking msbuild</b> in order to circumvent unfortunate designs found in Microsoft msbuild files (and later to found at work in Xamarin build files as well). In this post, we will go through the gotchas found, and we will see also <b>why Visual Studio should really improve the configuration manager</b> if they want to improve our developers experience.<br />
<a name='more'></a><br />
<h3>
Preliminaries</h3>
There are a couple of things to understand on how VisualStudio and msbuild are working with solution files and configuration. This is just a little overview about the key settings and how they affect your build. I found some good introduction about this in the post "<a href="http://www.galaktor.net/2011/04/targeting-platforms-in-visual-studio.html" target="_blank">Targeting Platforms in Visual Studio</a>" worth a read.<br />
<br />
If we look at a simple solution containing only a single project:<br />
<br />
<pre class="brush:csharp">Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 2012
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TestConsole", "TestConsole\TestConsole.csproj", "{56849035-CEF7-446D-AF0A-51EE9DC1DDB7}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{56849035-CEF7-446D-AF0A-51EE9DC1DDB7}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{56849035-CEF7-446D-AF0A-51EE9DC1DDB7}.Debug|Any CPU.Build.0 = Debug|Any CPU
{56849035-CEF7-446D-AF0A-51EE9DC1DDB7}.Release|Any CPU.ActiveCfg = Release|Any CPU
{56849035-CEF7-446D-AF0A-51EE9DC1DDB7}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
EndGlobal
</pre>
<br />
What we can see from the solution is that it defines:<br />
<br />
In <code>SolutionConfigurationPlatforms</code>, the mapping between solution configuration/platforms to project configuration/platforms. When you read the line :<br />
<pre class="brush:csharp"> Debug|Any CPU = Debug|Any CPU
</pre>
<br />
It means that the Solution configuration/platform <code>Debug|Any CPU</code> will map to the project configuration/platform <code>Debug|Any CPU</code>.<br />
<br />
The project configuration and platform are the actual values that will be used when using later the properties <code>Configuration</code> and <code>Platform</code> in the msbuild proj (csproj...etc.) as we can see it used by the <code>TestConsole.csproj</code> above:
<br />
<pre class="brush:xml">
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<PlatformTarget>AnyCPU</PlatformTarget>
<DebugSymbols>true</DebugSymbols>
</pre>
The solution also defines in the section <code>ProjectConfigurationPlatforms</code> the projects that will be build for each solution configuration/platform, as well as a mapping to the actual project configuration/platform.
In SharpDX, the configuration/platform in <a href="https://github.com/sharpdx/SharpDX/blob/master/SharpDX.sln#L127" target="_blank">SharpDX.sln</a> are configured like this:
<br />
<pre class="brush:csharp">
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|DIRECTX11_2 = Debug|DIRECTX11_2
Debug|Net20 = Debug|Net20
Debug|Net40 = Debug|Net40
Debug|Win8 = Debug|Win8
Debug|WP81 = Debug|WP81
Debug|WP8-ARM = Debug|WP8-ARM
Debug|WP8-x86 = Debug|WP8-x86
Release|DIRECTX11_2 = Release|DIRECTX11_2
Release|Net20 = Release|Net20
Release|Net40 = Release|Net40
Release|Win8 = Release|Win8
Release|WP81 = Release|WP81
Release|WP8-ARM = Release|WP8-ARM
Release|WP8-x86 = Release|WP8-x86
EndGlobalSection
</pre>
As you can see, we are just using different configuration/platforms in order to target multiple .NET framework, different DirectX version and specifics OSes. But surprisingly, if you are trying to use this kind of configuration in your solution, It will not work out of the box.
<br />
<br />
<h3>
Problem #1: Where is the solution platform?</h3>
<br />
By default, Visual Studio settings in C# is hiding the solution platform. Instead, what you will get is only the solution configuration:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUc7bT7X7YkCO1ZDr-hCuuRaHbcRMpCkQJ46-Ul7JZX6aQL4mwKQzDT0KKqWhTWeeOM9BAqsa9oYc23XIeQF6KdbTIYqFB7XujPtmP4T5DLmDrzZCNGY2uCZw5iNtN8Q0NoX17q05Vjrg/s1600/VSConfigSelector.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUc7bT7X7YkCO1ZDr-hCuuRaHbcRMpCkQJ46-Ul7JZX6aQL4mwKQzDT0KKqWhTWeeOM9BAqsa9oYc23XIeQF6KdbTIYqFB7XujPtmP4T5DLmDrzZCNGY2uCZw5iNtN8Q0NoX17q05Vjrg/s1600/VSConfigSelector.png" /></a></div>
<br />
<br />
This is really annoying, because if someone just open your solution, It will not realize that there are actually different platforms. The solution will just select the first defined platform.
In order to get back the solution platform selector in Visual Studio, you need to activate back the button by selecting on the right side of the solution configuration the drop-down button "Add or Select buttons":<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEIPXglBL1E1hn6RAZLTAV0fvFHAH9_lNW07tG40PctNuNx-RaBCiGuNCX2RoK9B2bqd7clM8O6YcfG94RHa6NZ3PMarqd-iiAqfW3ghq6HXEzaK71Bdf7fCImqgOeAqOo8YvIFUfPJMc/s1600/VSRestorePlatformSelector.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEIPXglBL1E1hn6RAZLTAV0fvFHAH9_lNW07tG40PctNuNx-RaBCiGuNCX2RoK9B2bqd7clM8O6YcfG94RHa6NZ3PMarqd-iiAqfW3ghq6HXEzaK71Bdf7fCImqgOeAqOo8YvIFUfPJMc/s1600/VSRestorePlatformSelector.png" height="480" width="640" /></a></div>
<br />
While I understand the ergonomic original reasons for hiding this button, in the era of multiple platform development, this should no longer be hidden and the default should show it. I hope that Visual Studio will fix this in a future release.<br />
<br />
<h3>
Problem #2: Project Platform semantic</h3>
<br />
This is the problem that made the refactoring of the SharpDX.sln quite laborious to hack. On the surface, solution platforms look nice. They provide a way to organize your project to target multiple platforms/configurations from the same solution. On the backside, it is not working as expected, mainly because <b>some msbuild files are interpreting the value of the project platform</b>.<br />
<br />
And this is where I would like to take the opportunity here to explain why <b>project platforms should have no semantic values for Visual Studio or Xamarin build files</b>. Project platforms should be considered as user defined platforms, they are a way to organize our project in whatever combinations and <b>these semantics should be owned by the developer of the project</b>.<br />
<br />
Unfortunately, Visual Studio msbuild files don't allow to use a custom project platform because they are expecting some specific platforms. For example, if you are developing a Windows Store Apps, you will find that a Windows Store Apps project won't compile if the platform is different from "Any CPU/x86/x64//Win32/arm"!. This is hardcoded in the file <code>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\AppxPackage\Microsoft.AppxPackage.Targets</code> line 1270 like this (Windows Phone platform and Xamarin are suffering the same problem):<br />
<pre class="brush: xml"><PropertyGroup>
<_ProjectArchitectureOutput>Invalid</_ProjectArchitectureOutput>
<_ProjectArchitectureOutput Condition="'$(Platform)' == 'AnyCPU'">neutral</_ProjectArchitectureOutput>
<_ProjectArchitectureOutput Condition="'$(Platform)' == 'x86'">x86</_ProjectArchitectureOutput>
<_ProjectArchitectureOutput Condition="'$(Platform)' == 'Win32'">x86</_ProjectArchitectureOutput>
<_ProjectArchitectureOutput Condition="'$(Platform)' == 'x64'">x64</_ProjectArchitectureOutput>
<_ProjectArchitectureOutput Condition="'$(Platform)' == 'arm'">arm</_ProjectArchitectureOutput>
</PropertyGroup>
</pre>
Using directly the <code>Platform</code> from a core VisualStudio msbuild file is a mistake (same for Configuration, that is used in some Visual Studio msbuild targets), as it is forcing the original solution to use only these platforms. Instead, build files from Visual Studio should use a property that can be redefined by the project (like the property <code>PlatformTarget</code> that is used by the C# compiler). We should have a way to redefine the mapping in whatever way we would like.
In other words, Solution platform and configurations should be fully owned by the developer of the solution. Their semantics are project specific and Visual Studio should allow us to define the remapping to a target platform (like <code>AnyCPU</code>) in our project like this:
<br />
<pre class="brush:xml">
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|ThisIsMyConfig' ">
<PlatformTarget>AnyCPU</PlatformTarget>
...
</pre>
Fortunately, there is a hack to manage this, though it is not completely safe. By default, the properties <code>Platform</code> and <code>Configuration</code> are immutable in msbuild, because they are considered as global properties passed to msbuild, so they cannot be modified. But there is a way to override the platform "ThisIsMyConfig" to "AnyCPU" for some specific build (like WindowsStoreApps).
In SharpDX, this is made possible by the target "SharpDXForcePlatform" as can be seen in this <a href="https://github.com/sharpdx/SharpDX/blob/master/Build/SharpDX.PreSettings.targets#L353" target="_blank">file</a>. In order to work, the trick is:
<br />
<ul>
<li>Add a target that will be executed automatically whenever there is a build. This is done by declaring a msbuild project with the attribute <code>InitialTargets="YourTargetToForcePlatform"</code></li>
<li>In the <code>YourTargetToForcePlatform</code>, we can override the <code>Platform</code> property programmatically (they are mutable only when using this trick from a target). In the following code, we are remapping the <code>Platform</code> <b>Win8</b> to <code>AnyCPU</code> like this:
<pre class="brush:xml">
<Target Name="SharpDXForcePlatform">
<!--
Windows 8 App Store => AnyCPU
Windows Phone 8.1 => AnyCPU
-->
<CreateProperty Condition=" '$(Platform)' == 'Win8' or '$(Platform)' == 'WP81'" Value="AnyCPU">
<Output
TaskParameter="Value"
PropertyName="Platform" />
</CreateProperty>
</pre>
</li>
</ul>
This way, when the build start, the <code>Platform</code> property is correctly setup for the platform being compiled. Beware that the property <code>Platform</code> used outside a target (in property groups...etc.) is still linked to the original semantic which is actually good. But if a Visual Studio build is using the property <code>Platform</code> outside a target, this trick will not work.<br />
<br />
So bottom line of this problem is that Visual Studio builds should really take care of this and avoid forcing any semantic for the configuration/platform. Without this, we are forced to use the hack described above or worst, to duplicate the solution (this was the case for SharpDX, which made the full build quite a pain).<br />
<br />
<br />
<br />
<h3>
Problem #3: The unwanted Mixed Platforms</h3>
When you are using custom platforms names, and you want to add a new project to your solution, you will most likely end-up with a new solution platform Mixed Platforms. This is really annoying when we are already dealing with multiple platforms, we don't want Visual Studio to add a useless platform. The solution is to remove it by hand in the .sln, but we should not have to do this. At worst Visual Studio should ask the developer "Do you really want to add a new mixed platform to your solution?", at best, remove this Mixed Platforms.<br />
<br />
<br />
<h3>
Problem #4: The Configuration Manager</h3>
<br />
When managing several platforms with several dozens of projects, the configuration manager is a real pain to use, and we are always forced to edit the sln by hand and perform some regexp replace on the file to cleanup it or to fix it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWXrF1JPn3Tkw7E4ZWCp9Vc0AifT2k8Fd8hZcpgRXClIsadjQL0qfUZ_ag-YC-MjJrIqbXxbHT0cuEEiA2UOwb2qr4qh7uPtbtE4jYquYUsnBEVeuglDlnhx4WrqYn5Kt0zqPozd4vVFY/s1600/VSConfigurationManager.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWXrF1JPn3Tkw7E4ZWCp9Vc0AifT2k8Fd8hZcpgRXClIsadjQL0qfUZ_ag-YC-MjJrIqbXxbHT0cuEEiA2UOwb2qr4qh7uPtbtE4jYquYUsnBEVeuglDlnhx4WrqYn5Kt0zqPozd4vVFY/s1600/VSConfigurationManager.png" height="402" width="640" /></a></div>
<br />
<br />
<br />
There are lots of issues with the current Configuration Manager:<br />
<ul>
<li>The <b>window is not resizable </b>! If you have more than 12 projects in your solution, you are good to use the scrollview quite a lot.</li>
<li>It is <b>not possible to have a global view </b>of all your projects and which one is activated for which platforms...etc. Considering that you need to check (Debug AND Release) x number of
platforms, and you have go around for a while by clicking, waiting,
clicking, scrolling... a nightmare!</li>
<li>It is <b>not possible to bulk edit your projects</b>. You have to go though each single project, single click, dropdowns...etc. for each projects.</li>
<li><b>Switching configuration or platform is slow</b> when you have lots of projects (or some custom .targets). I don't understand why Visual Studio seems to reevaluate all the projects, so it can take 2-3 seconds when switching the configuration/platform while everything should be already accessible from memory (both solution and projects)</li>
</ul>
<br />
<h3>
vNext</h3>
Whoever has done some cross platform development (even just inside .NET, by targeting different .NET framework) with Visual Studio will most likely have struggled with the issues describe above.<br />
<br />
With the rise of Xamarin more tightly integrated into Visual Studio, more development targeting all Windows eco-system (Windows Desktop, Windows AppStore, Windows Phone Store) and Android/iOS, all these issues should be really fixed to improve our productivity. Fingers crossed for VS2014 if someone at the Visual Studio team is reading this!<br />
<br />
How do you manage these issues in your projects? Do you have any other ideas to improve the situation when targeting multiple platforms in Visual Studio?<br />
<br />
<span style="font-size: x-small;">PS: I will have to double check whether there is some uservoice or connect bugs for the issues described in this post. If you have any link already, I'm interested!</span> <br />
<br />
<br />xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com2tag:blogger.com,1999:blog-1076643699683521890.post-22976299193709726102013-04-20T03:21:00.000+11:002013-04-20T23:53:01.542+11:00Advanced HLSL II: Shader compound parametersA very short post in the sequel of my previous post "<a href="http://code4k.blogspot.com/2011/11/advanced-hlsl-using-closures-and.html" target="_blank">Advanced HLSL using closures and function pointers</a>", there is again a little neat trick by using the "class" keyword in HLSL: It is possible to use a class to regroup a set of parameters (shader resources as well as constant buffers) and their associate methods, into what is called a <b>compound parameter</b>. This feature of the language is absolutely not documented, I discovered the name "compound parameter" while trying to hack this technique, as the HLSL compiler was complaining about a restriction about this "compound parameter". So at least, It seems to be implemented up to the point that it is quite usable. Let's see how we can use this...<br />
<a name='more'></a><br />
<h3>
Group of input parameters in shaders, the usual way</h3>
Suppose the following code (not really useful):
<br />
<br />
<pre class="brush: hlsl" name="code">// Shader Resources
SamplerState PointClamp;
// First set of parameters
// -----------------------
Texture2D<float> DepthBuffer;
float2 TexelSize;
// Associated methods with these parameters
float SampleDepthBuffer(float2 texCoord, int2 offsets = 0)
{
return DepthBuffer.SampleLevel(PointClamp, texCoord + offsets * TexelSize, 0.0);
}
// Second set of parameters
// ------------------------
Texture2D<float> DepthBuffer1;
float2 TexelSize1;
// Associated methods with these parameters
float SampleDepthBuffer1(float2 texCoord, int2 offsets = 0)
{
return DepthBuffer1.SampleLevel(PointClamp, texCoord + offsets * TexelSize1, 0.0);
}
float4 PSMain(float2 texCoord: TEXCOORD) : SV_TARGET
{
return float4(SampleDepthBuffer(texCoord, int2(1, 0)), SampleDepthBuffer1(texCoord, int2(1, 0)), 0, 1);
}
</pre>
<br />
What we have is some parameters that are grouped, for example
<br />
<ul>
<li>A resource DepthBuffer</li>
<li>A TexelSize that gives the size of a texel in uv coordinates for the previous textures (float2(1/width, 1/height))</li>
<li>A method "SampleDepthBuffer" that will sample the depth buffer.</li>
</ul>
And this set of parameters is duplicate with another set with just the postfix number "1".
We need to duplicate the code here. Though of course, as usual there are some workaround
<br />
<ul>
<li>Either by using the preprocessor and token pasting: this approach is often used, but It means that you have a code that is sometimes less readable, especially if you have to embed a function in a #define.</li>
<li>For the methods SampleDepthBuffer, It could be possible to rewrite the signature to accept a Texture2D as well as a TexelSize as a parameter. Of course, if this function was using more textures, more parameters, we would have to pass them all by parameters...</li>
</ul>
The generated code produced by fxc.exe HLSL compiler is like this:
<br />
<pre style="color: green; font-size: 10px;">//
// Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
//
//
// fxc /Tps_5_0 /EPSMain test.fx
//
//
// Buffer Definitions:
//
// cbuffer $Globals
// {
//
// float2 TexelSize; // Offset: 0 Size: 8
// float2 TexelSize1; // Offset: 8 Size: 8
//
// }
//
//
// Resource Bindings:
//
// Name Type Format Dim Slot Elements
// ------------------------------ ---------- ------- ----------- ---- --------
// PointClamp sampler NA NA 0 1
// DepthBuffer texture float 2d 0 1
// DepthBuffer1 texture float 2d 1 1
// $Globals cbuffer NA NA 0 1
//
//
//
// Input signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------ ------
// TEXCOORD 0 xy 0 NONE float xy
//
//
// Output signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------ ------
// SV_TARGET 0 xyzw 0 TARGET float xyzw
//
</pre>
<pre style="font-size: 10px;">ps_5_0
dcl_globalFlags refactoringAllowed
dcl_constantbuffer cb0[1], immediateIndexed
dcl_sampler s0, mode_default
dcl_resource_texture2d (float,float,float,float) t0
dcl_resource_texture2d (float,float,float,float) t1
dcl_input_ps linear v0.xy
dcl_output o0.xyzw
dcl_temps 1
mad r0.xyzw, cb0[0].xyzw, l(1.000000, 0.000000, 1.000000, 0.000000), v0.xyxy
sample_l_indexable(texture2d)(float,float,float,float) r0.x, r0.xyxx, t0.xyzw, s0, l(0.000000)
sample_l_indexable(texture2d)(float,float,float,float) r0.y, r0.zwzz, t1.yxzw, s0, l(0.000000)
mov o0.xy, r0.xyxx
mov o0.zw, l(0,0,0,1.000000)
ret
// Approximately 6 instruction slots used
</pre>
<br />
When we have to deal with lots of parameters that are grouped, and these groups need to be duplicated with their associated methods, It becomes almost impossible to maintain a clean and reusable HLSL code.
Fortunately, the "class" keyword is here to the rescue!
<br />
<br />
<h3>
Shader input compound parameters container, the neat way</h3>
Let's rewrite the previous code using the keyword "class":<br />
<br />
<pre class="brush: hlsl" name="code">SamplerState PointClamp;
// Declare a container for our set of parameters
class TextureSet
{
Texture2D<float> DepthBuffer;
float2 TexelSize;
float SampleDepthBuffer(float2 texCoord, int2 offsets = 0)
{
return DepthBuffer.SampleLevel(PointClamp, texCoord + offsets * TexelSize, 0.0);
}
};
// Define two instance of compound parameters
TextureSet Texture1;
TextureSet Texture2;
float4 PSMain2(float2 texCoord: TEXCOORD) : SV_TARGET
{
return float4(Texture1.SampleDepthBuffer(texCoord, int2(1, 0)), Texture2.SampleDepthBuffer(texCoord, int2(1, 0)), 0, 1);
}
</pre>
<br />
And the resulting compiled HLSL is slightly equivalent:<br />
<br />
<pre style="color: green; font-size: 10px;">//
// Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
//
//
// fxc /Tps_5_0 /EPSMain2 test.fx
//
//
// Buffer Definitions:
//
// cbuffer $Globals
// {
//
// struct TextureSet
// {
//
// float2 TexelSize; // Offset: 0
//
// } Texture1; // Offset: 0 Size: 8
// Texture: t0
//
// struct TextureSet
// {
//
// float2 TexelSize; // Offset: 16
//
// } Texture2; // Offset: 16 Size: 8
// Texture: t1
//
// }
//
//
// Resource Bindings:
//
// Name Type Format Dim Slot Elements
// ------------------------------ ---------- ------- ----------- ---- --------
// PointClamp sampler NA NA 0 1
// Texture1.DepthBuffer texture float 2d 0 1
// Texture2.DepthBuffer texture float 2d 1 1
// $Globals cbuffer NA NA 0 1
//
//
//
// Input signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------ ------
// TEXCOORD 0 xy 0 NONE float xy
//
//
// Output signature:
//
// Name Index Mask Register SysValue Format Used
// -------------------- ----- ------ -------- -------- ------ ------
// SV_TARGET 0 xyzw 0 TARGET float xyzw
//
</pre>
<pre style="font-size: 10px;">ps_5_0
dcl_globalFlags refactoringAllowed
dcl_constantbuffer cb0[2], immediateIndexed
dcl_sampler s0, mode_default
dcl_resource_texture2d (float,float,float,float) t0
dcl_resource_texture2d (float,float,float,float) t1
dcl_input_ps linear v0.xy
dcl_output o0.xyzw
dcl_temps 1
mad r0.xy, cb0[0].xyxx, l(1.000000, 0.000000, 0.000000, 0.000000), v0.xyxx
sample_l_indexable(texture2d)(float,float,float,float) r0.x, r0.xyxx, t0.xyzw, s0, l(0.000000)
mov o0.x, r0.x
mad r0.xy, cb0[1].xyxx, l(1.000000, 0.000000, 0.000000, 0.000000), v0.xyxx
sample_l_indexable(texture2d)(float,float,float,float) r0.x, r0.xyxx, t1.xyzw, s0, l(0.000000)
mov o0.y, r0.x
mov o0.zw, l(0,0,0,1.000000)
ret
// Approximately 8 instruction slots used
</pre>
<br />
There are a couple of things to highlight:
<br />
<ul>
<li>The main difference is the packing of constant buffer variable is done separately as they will be packed together - as a struct and aligned on a float4 boundary. So in this specific case, the two floats TexelSize cannot be swizzled/merged (if they were float4, the code would be strictly equivalent). So we need to be aware and careful about this behavior.</li>
<li>Input resources are nicely prefixed by their compound parameter name, like "Texture1.DepthBuffer" or "Texture2.DepthBuffer", so it is also really easy to access them when using named resource bindings in an effect. Note that a resource declared but unused inside a compound parameter will occupy a slot register without using it (This is not a big deal, as there is almost the same kind of behavior when using array of resources)</li>
<li>We can still enclose "TextureSet Texture1" into a constant buffer declaration, the variable defined inside TextureSet for the Texture1 instance will correctly end-up in the corresponding constant buffer.</li>
<li>Global variable are accessible from methods defined in a compound parameter (for example PointClamp SamplerState used by the SampleDepthBuffer method)</li>
<li><b>Compound parameters can only be compiled using SM5.0 </b>(unlike the previous post about the closures).</li>
</ul>
This is really a handy feature that could help to better organize some of our shaders. It's always surprising to still discover this kind of syntax constructions accessible from the current HLSL compiler. Let me know if you find any issues using this trick!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com1tag:blogger.com,1999:blog-1076643699683521890.post-49978386071279746842012-08-08T18:59:00.001+11:002012-11-06T02:54:17.632+11:00Going Native 2.0, The future of WinRTIn the recent years, we have seen lots of fuzz about the return of “Going native” after the managed era popularized by Java and .NET. When WinRT was revealed last year, there was some shortsighted comments to claim that “.NET is dead” and to glorify the comeback of the C++, the true and only real way to develop an application, while at the same time, JIT was being more and more introduced in the scripted world (JavaScript being one of the most prominent JIT user). While in the end, everything is going native anyway - the difference being the length of the path to go native and how much optimized it will be - the meaning of the “native” word has slightly shifted to be strongly and implicitly coupled with the word “performance”. Even being a strong advocator for managed language, the performance level is indeed below a well written C++ application, so should we just accept this fact and get back to work with C++, with things like WinRT being the backbone of the interop? To tell you the truth, I want .NET to die and this post is about why and for what.<br />
<br />
<h3>
The Managed Era</h3>
<br />
Let’s just begin by revisiting recent history of managed development that will highlight current challenges. Remember the Java slogan? “write once runs everywhere”, it was the introduction of a paradigm where a complete “safe” single language-stack based on a virtual machine associated with a large set of API would allow to easily develop an application and target any kind of platforms/OS. It was the beginning of the “managed” era. While Java has been quite successfully adopted in several development industries, it was also quite rejected by lots of developers that were aware of memory management caveats and the JIT not being as optimized as it should be (though they did some impressive improvements over the years) with also a tremendous amount of bad design choice, like the lack of native struct, unsafe access or the route to go native through JNI extremely laborious and inefficient (and even <a href="http://marxsoftware.blogspot.jp/2011/10/javaone-2011-from-java-se-2012-to-java.html">recently</a>, that they were considering to get rid off all native types and make everything an object, what a terrible direction!). <br />
<a name='more'></a><br />
Java failed also in the heart of his slogan: it was in fact not possible to embrace in a single unified API all the usage of each target platforms/OS, leading to things like Swing, not what can be called an optimal UI framework. Also, from the beginning, Java was only design with a single language in mind, though lots of people found JIT/bytecode as an opportunity to port scripting languages to Java JVM. <br />
<br />
In the meantime of early Java, Microsoft that tried to enter the Java market by integrating some custom language extensions (with the end story we know) and finally came with their own managed technology, which was in several aspects better conducted and designed: from the ground bytecode, unsafe construct, native interop, lightweight but very efficient JIT + NGEN, C# rapid language evolution, C++/CLI... etc, taking multiple language interop into account from the beginning and without the burden of the Java slogan (though Silverlight on MacOS or Moonlight were a good try). <br />
<br />
Both systems share a similar managed monolithic stack: metadata, bytecode, JIT and GC are tightly coupled. Also performance wise, it is far from being perfect: the JIT is implying a startup cost and the executing code is not as fast as it should mainly because:<br />
<ol>
<li>The JIT is performing poor optimization compare to full C++ -O2, because it needs to be fast when generating code (also, unlike Java hotspot JVM, .NET JIT is not able to hot swap existing JIT code by a better optimized code)</li>
<li>Managed types, like Array access are always checking bounds (apart for simple loops where the JIT can suppress the check if the for-limit is less or equal the array’s length)</li>
<li>GC can pause all threads to collect objects (though new GC in 4.5 made some <a href="http://blogs.msdn.com/b/dotnet/archive/2012/07/20/the-net-framework-4-5-includes-new-garbage-collector-enhancements-for-client-and-server-apps.aspx">improvements</a>) which can cause unexpected slow down in an application.</li>
</ol>
But even with this performance deficiency, a managed eco-system with its comprehensive Framework is the king of productivity and language interop, with a descent overall performance for all languages running inside it. The apogee of the managed era was probably around the launch of Windows Phone and Visual Studio 2010 (using WPF for its rendering, though WPF is also built on top of lots of native code), where managed languages were the only authorized way to develop an application. That was not the best thing that could happen, considering the long list of pending issues with .NET performance, enough to stimulate all the “native coders” to strike back, and they were absolutely in their rights.<br />
<br />
It turns out that somewhat it signs the "decline" of .NET. I don’t know much about Microsoft organization internals, but what is commonly reported is that there is some serious competition between divisions, good or bad, but for .NET, for the past few years, Microsoft seemed to running out of gas (for example, almost no significative improvements in the JIT/NGEN, lots of pending request for performance improvements, including things like SIMD that were asked for a long time), and my guess is that the required changes could only take place in a global strategy, with deep support and commitment from all divisions. <br />
<br />
In the mean time, Google was starting to push its NativeClient technology, allowing to run sandboxed native code from the browser. Last year, in this delirium trend of going native, Microsoft revealed that even HTML5 implemented in next IE was going native! Sic.<br />
<br />
In "<a href="http://herbsutter.com/2012/04/02/reader-qa-when-will-better-jits-save-managed-code/">Reader Q&A: When will better JITs save managed code?</a>" Herb Sutter, one of the "Going Native" evangelist, provides some interesting insights about what the "Going Native" philosophy is thinking about JIT, with lots of inaccurate facts, but lets just focus on the main one : Even if JIT could improve in the future, managed languages made such a choice of safety over performance, that they are intrinsically doomed to not play in the big leagues. Miguel de Icaza posted a response about it in "<a href="http://tirania.org/blog/archive/2012/Apr-04.html">Can JITs be faster?</a>" and he explained lots of relevant things about why some of Herb Sutter statements were misleading.<br />
<br />
Then WinRT came here to somewhat smooth the lines. By taking part of the .NET philosophy (metadata and some common “managed” types like strings and arrays) and the good old COM model (for a common denominator of native interop), WinRT is trying to solve the problem of language interoperability outside the CLR world (thus without the performance penalties for C++) and to provide a more “modern” OS API. Is this the definitive answer, the one that will rule them all? So far, not really, it is on the direction of the certain convergence that could lead to great things, but it is still uncertain that it will take the right track. But what could be this “right track”?
<br />
<br />
<h3>
Going native 2.0, Performance for All</h3>
<br />
Though safety rules can have a negative impact on performance, managed
code is not doomed to be run by poor JIT compiler (For example, Mono is
able to run C# code natively compiled through LLVM on iOS/Linux) and it
would be fairly easy to extend the bytecode with more "unsafe" levels to provide fine grained performance speedup (like suppressing array bounds checking...etc.). <br />
<br />
But the first problem that can be currently identified is the lack of a strong cross-language compiler infrastructure, this is ranging from the compiler used in IE10 Javascript JIT, to the .NET JIT and NGEN compilers or into the Visual C++ compilers (to name a few), all using different code for almost the same kind of laborious and difficult problem of generating efficient machine code. Having a single common compiler is a very important step to provide a high performance code accessible from all languages.<br />
<br />
Felix9 on Channel9 <a href="http://channel9.msdn.com/Forums/Coffeehouse/MS-working-on-a-same-compiler-for-C-AND-C--Not-in-incubation-but-for-production-">found</a> that Microsoft could be actually working on this problem, so that's a good news, but the problem of the "performance for all" is a small part of a bigger picture. In fact the previous mentioned "right track" is a broader integrated architecture, not only an enhanced <a href="http://llvm.org/">LLVM stack</a>, but baked by Microsoft's experience in several fields (C++ compiler, JIT, GC, metadatas... etc), a system that would expose a <b>completely externalized and modularized “CLR”</b> composed of:<br />
<br />
<ul>
<li><b>An intermediate mid level language</b>, entirely queriable/reflectionnable, very similar to LLVM IR or .NET bytecode, defining common datatypes (primitives, string, array... Etc). An API similar to System.Reflection.Emit should be available. Vectorized types (SIMD) should be first class types as int or double are. This IL code should not be limited to CPU target usage, but should allow GPU computing (similar to AMP) : it should be possible to express HLSL bytecode with this IL, with the benefits to leverage on a common compiler infrastructure (see following points). Typeless IL should also be possible to allow dynamic languages to be expressed more directly.</li>
<li><b>A dynamic linked library/executable, like assemblies in .NET, providing metadatas, IL code</b>, query/reflection friendly. When developing, code should be linked against assemblies/IL code (and not against crappy C/C++ headers).</li>
<li><b>An IL to native code compiler</b>, which could be integrated in a JIT, an offline or a cloud compiler, or a mixed combination. This compiler should provide vectorization whenever target platform support it. IL code would be compiled to native code at install/deploy time, based on the target machine architecture (at dev time, it could be done after the whole application has been compiled to IL). The compiler stages should be accessible from an API and offer extension points as much as possible (providing access to IL to IL optimization, or to provide pluggable IL to native code transform). The compiler would be responsible to perform global program optimization at deploy time (or at runtime for JIT scenarios). Optimizations options should range from fast compilation (like JIT) to aggressive (offline, or hot swap code in a JIT). A profile of the application could also be used to automatically tune localized optimizations. This compiler should support advanced JIT scenarios, like dynamic hotspot analysis and On Stack Replacement (aka OSR, allowing heavy computation code to be replaced at runtime by a better optim code), unlike current .NET JIT that only compiles a method on a 1st run. This kind of optimization are really important in dynamic scenarios where type inference is sometimes discovered later (like Javascript).</li>
<li><b>An extensible allocator/memory component, allowing concurrent allocators, where the Garbage Collector/GC would be one implementation</b>, though a major part of applications would use it to manage most of their lifecycle objects, leaving the most performance critical objects to be managed by other allocator schemes (like reference counting scenarios used by COM/WinRT). There is no restrictions to use different allocator models in a same application (and this is already what's happening when in a .NET application we need to deal with native interop to allocate objects using OS functions).</li>
</ul>
The philosophy is very similar to a CLR stack, however it doesn't force an application to be ran by a JIT compiler (yes there is NGEN in .NET, but it was designed for startup reasons, not for high performance reasons, plus it is a black box only working on assemblies installed into the GAC) and it allows mixed memory allocation GC/non-GC scenarios. <br />
<br />
In this system, full native interoperability between languages would then be straightforward without sacrifying performance over simplicity and vice-verca. Ideally, an OS should be built from the ground up with such a core infrastructure. This is what was (is?) probably behind a project like <a href="http://www.zdnet.com/blog/microsoft/microsoft-codename-redhawk-lives-in-windows-8/9233">Redhawk</a> (for the compiler part), or <a href="http://www.zdnet.com/blog/microsoft/goodbye-xp-hello-midori/1466">Midori</a> (for the OS part), in such an integrated system, probably only drivers would require some kind of unsafe behaviors.<br />
<br />
[<b>Update 9 Aug 2012</b>: Felix9 again <a href="http://channel9.msdn.com/Forums/Coffeehouse/MS-patent-Optimizer-as-an-AppStore-Service-Cloud-JIT-/c539c486cf914d3ab718a0a700fb473e">found</a> that an intermediate bytecode, more low level than MSIL .NET bytecode, called MDIL could be already in used, and that could be the intermediate bytecode mentioned just above, though looking at the related patent "<a href="http://www.freepatentsonline.com/y2011/0258616.html">INTERMEDIATE LANGUAGE SUPPORT FOR CHANGE RESILIENCE</a>", there are some native x86 registers in the specs that don't fit well with an architecture independent bytecode. Maybe they would keep MSIL as-is and leverage on a lower level MDIL. We will see.].<br />
<br />
So what WinRT is tackling in this big picture? Metadatas, a bit of sandboxes API and an embryo of interoperability (through common datatypes and metadatas), as we can see, not so much, a basic COM++. And as we can obviously realize, <b>WinRT is not able to provide advanced optimizations in scenarios where we use a WinRT API</b>: for example, we cannot have a plain structure that can expose inlinable methods. Every method calls in WinRT are virtual calls, forced to go through a vtable (and sometimes several virtual calls are needed, when for example a static method is used), so even a simple property get/set will go through a virtual call. This is clearly inefficient. It looks like WinRT is only targeting coarse level API, leaving all the fine grained level API at the mercy of performance heterogeneity, restricting common scenarios where we want to access high performance code <i>everywhere</i>, without going through a layer of virtual calls and non-inlinable code. Using an extended COM model is not what we can call “Building the Future”. <br />
<br />
<h3>
Productivity and Performance for C# 6.0 </h3>
<br />
A language like C# would be a perfect candidate in such a modular CLR system, and could be mapped easily to the previous intermediate bytecode. Though to efficiently use such a systen, C# should be improved on several aspects: <br />
<ul>
<li><b>More unsafe power</b> where we could turn off
“managed” behaviors like array access checking (kind of “super unsafe
mode”, where we could possibly use CPU pre-caching instructions before
accessing next array elements, kind of "advanced" stuff impossible to do
with current managed arrays without using unsupported tricks)</li>
<li><b>A configurable new operator</b> that would integrate different allocator schemes. </li>
<li><b>Vectorized types</b> (like HLSL float4) should be added to the core types. This has been asked for a long time (with ugly patches in XNA WP to "solve" this problem).</li>
<li><b>Lightweight interop to native code</b> (in the case we would still be calling native code from C# unlike in an integrated OS): current manage to unmanaged transition is costly when calling native methods even without any "fixed" variables. An unsafe transition should be possible without the burden of the current x86/x64 prologue/epilogue of the unmanaged transition generated by current .NET JIT.</li>
</ul>
From a general language perspective, not strictly related to performance, there are lots of small area that would be important to be addressed as well:<br />
<ul>
<li><b>Generics everywhere</b> (in constructors, in implicit conversions) with more
advanced constructs (contracts on operators... etc), closer to C++
template versatility but safer and less cluttered.</li>
<li><b>Struct
inheritance and finalizers</b> (to allow lightweight code to be executed on exit of a method, without going through the cumbersome "try/finally" or "using" patterns).</li>
<li><b>Add more MetaProgramming</b>: allow static method extensions (not only for "this"), allow class mixin (mixin the content of a class inside another, usefull for things like math functions), allow modification of class/types/methods construction at compile time (for example, methods that would be called at compile time to add method/properties to a class, very similar to <a href="http://stackoverflow.com/questions/1630815/why-isnt-the-eigenclass-equivalent-to-self-class-when-it-looks-so-similar">eigenclass in Ruby meta-programming</a> instead of using things like T4 template code generation), more extensively, allow DSL like syntax language extensions at several points into the C# parser (Roslyn doesn't provide currently any extension point inside the parser) so that we could express language extensions in C# as well (for example, instead of having Linq syntax hardcoded, we should be able to write it as an extension parser plugin, fully written in C#). [Edit] I have posted a discussion "<a href="http://social.msdn.microsoft.com/Forums/en-US/roslyn/thread/8e930e49-cc3d-4434-9a94-0dc7ff98d2ed" target="_blank">Meta-Programming and parser extensibility in C# and Roslyn</a>" about what is intended behind this meta-programming thing at the Microsoft Roslyn forum. Check it out![/Edit]</li>
<li><b>A builtin symbol or link type where we could express a link to a language object </b>(a class, a property, a method) by using a simple construction like: <code><span style="color: blue;">symbol</span> LinkToMyMethod = @MyClass.MyMethod;</code> instead of using Linq expressions (like (myMethod) => MyMethod inside MyClass). This would make more robust code using INotifyPropertyChanged or simplify all property based systems like WPF (which is currently an ugly duplication of the method definition).</li>
</ul>
Bottom line, is that <b>there is less to
add to C# than there is to remove from C++ to fully leverage on such a
system and to greatly improve developer’s productivity, again without
burning efficiency</b>. One could argue that C++ already offers all of this and much more, but this is exactly why C++ is so much cluttered (syntax wise) and
dangerous for the vast majority of developers. It allows
unsafe everywhere, while unsafe code is always localized in an
application (and is always source of memory corruption, so it is much
easier to fix if they are clearly identified and strictly localized in
the code, same than using asm keyword in non standard C/C++). It is
easier and safer to track exceptional usages in a large codebase than to have it allowed everywhere. <br />
<br />
<h3>
Next? </h3>
<br />
We can hope that Microsoft took a top-down approach, by addressing unified OS API for all languages and simple interoperability first, and that they will introduce these more advanced features in later version of their OS. But this is an ideal expectation and it will be interesting to follow if Microsoft will effectively challenge this. Even if It was recently revealed that WP8 .NET applications would
benefit some Cloud compilers, so far, we don't know much about it:
Is it just a repackaging of NGEN (which is again, not performance
oriented, generating code very similar to current JIT) or a non public
RedHawk compiler?<br />
<br />
Microsoft has lots of gold in their backyard, with years of advanced native code compilations with their C++ compiler, JIT, GC, and all the related R&D projects they have...<br />
<br />
So to summarize this post: .NET must die to a better integrated, performance oriented, common runtime where the managed (safety/productivity) vs native (performance) is no longer a border, and this should be a structural part of next WinRT architecture evolution.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com20tag:blogger.com,1999:blog-1076643699683521890.post-4409880640002812302011-11-24T23:09:00.001+11:002012-08-18T16:00:42.474+11:00Advanced HLSL using closures and function pointersShader languages like HLSL, Cg or GLSL are nowadays driving the most powerful processors in the world, but if you are developing with them, you may have been already a little bit frustrated by one of their expressiveness limitations: the common problem of abstraction and code reuse. In order to overcome this problem, solutions so far were mostly using a glue combination of #define/#include preprocessors directives in order to generate combinations of code, permutation of shaders, so called UberShaders. Recently, this problem has been addressed, for HLSL (new in Direct3D11), by providing the concept of <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff471421%28v=VS.85%29.aspx">Dynamic Linking</a>, and for GLSL, the concept of <a href="http://www.g-truc.net/post-0269.html">SubRoutines</a>, For Direct3D11, the new mechanism has been only available for Shader Model 5.0, meaning that even if this could greatly simplified the problem of abstraction, It is unfortunately only available for Direct3D11 class graphics card, which is of course a huge limitation...<br />
<br />
But, here is the good news: While the classic usage of dynamic linking is not really possible from earlier version (like SM4.0 or SM3.0), I have found<b> an interesting hack to bring some kind of closures and functions pointers to HLSL</b>(!). This solution doesn't involve any kind of preprocessing directive and is able to <b>work with SM3.0 and SM4.0</b>, so It might be interesting for folks like me that like to abstract and reuse the code as often as possible! But let's see how It can be achieved...<br />
<a name='more'></a><br />
<br />
<h3>
A simple problem of abstraction and code reuse in HLSL</h3>
<br />
I have been working recently at my work on a GPU implementation of a versatile perlin/simplex/fbm/turbulence noise in HLSL. While some of the individual algorithm are pretty simples, it is often common to use several permutations of those functions in order to produce some nice noise and turbulences functions (like the <a href="http://code4k.blogspot.com/2010/08/making-of-ergon-4k-pc-intro.html#wormlava">worm-lava texture I did for Ergon 4k intro</a>). Thus, they are an ideal candidate to demonstrate the use of closures and functions pointers. I won't explain here the basic principle of <a href="http://www.noisemachine.com/talk1/">perlin</a> and <a href="http://en.wikipedia.org/wiki/Fractional_Brownian_motion">fbm</a> noise generation to focus on the problem of code reuse in HLSL.<br />
<br />
<br />
Here is a simplified version of a Turbulence Noise implemented in a Pixel Shader:
<br />
<br />
<pre class="brush: hlsl" name="code">float PerlinNoise(float2 pos){
....
}
float AbsNoise(float2 pos) {
return abs(PerlinNoise(pos));
}
float FBMNoise(float2 pos) {
float value = 0.0f;
float frequency = InitialFrequency;
float amplitude = 1.0f;
// Classic FBM loop
for ( int i=0; i < Octaves; i++ )
{
float noiseValue = AbsNoise(pos);
value += amplitude * noiseValue;
frequency *= Lacunarity;
amplitude *= Amplitude;
}
return value;
}
// Turbulence noise:
// Fbm + Abs + Perlin
float TurbulenceAbsPerlinNoisePS(float4 pos : SV_POSITION, float2 texPos : TEXCOORD0)
: SV_Target
{
return FBMNoise(texPos);
}
</pre>
<br />
The problem with the previous code is that if we want to change the code behind <b>AbsNoise</b> called from FBMNoise (for example, apply cos/sin on the coordinates, or use of a simplex noise instead of the old Perlin Noise), <b>we would have to duplicate the FBMNoise function to call the other function</b>. Of course, we could use the preprocessor to inline the code, but It would end up in something less readable, less debuggable, error prone...etc.<br />
<br />
Another example: Ken Perlin introduced some really cool functions to modify the noise, like the famous <a href="http://www.noisemachine.com/talk1/23.html">marble effect</a>:<br />
<br />
<pre class="brush: hlsl" name="code">static float stripes(float x, float f) {
float t = .5 + .5 * sin(f * 2*PI * x);
return t * t - .5;
}
float MarbleNoise(float2 pos) {
return stripes(pos.x + 2 * FBMNoise(pos), 1.6f);
}
</pre>
<br />
But wait! The MarbleNoise function could even be used in place of the AbsNoise function, in order to get another noise effect. So we could have a marble function calling a FBM... but we could also have a marble function called by a FBM... or both... ugh... so as we can see, It is possible to permute those functions to generate interesting patterns, but unfortunately, the shading language doesn't provide us a way to make those functions pluggable!... Almost! In fact, there is a small breach in the HLSL language and we are going to use it!<br />
<br />
<br />
<h3>
Introduction to Dynamic Linking in HLSL</h3>
<br />
So as I said in the introduction, Direct3D11 has introduced the concept of dynamic linking. I suggest the reader to go to an explanation on msdn "<a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff471421%28v=VS.85%29.aspx">Interfaces and classes</a>". Basically, the main feature introduced in the HLSL language is a bit of Object Oriented Programming (OOP) in order to address the problem of abstraction: Now HLSL has the <b>class</b> and <b>interface </b>keyword. But they were mainly introduced for dynamic linking of a shader, and as I said, dynamic linking is only available with SM5.0 profile.<br />
<br />
<br />
<pre class="brush: hlsl" name="code">// An interface describing a light
interface ILight {
float3 ComputeAmbient(...);
float3 ComputeDiffuse(...);
float3 ComputeSpecular(...);
};
// A 1st implem of the ILight interface
class MyModelLight1 : ILight {
float3 ComputeAmbient(...) {
...
return color;
}
...
};
// A 2ns implem of the ILight interface
class MyModelLight2 : ILight {
float3 ComputeAmbient(...) {
...
return color;
}
...
}
// The variable through which we are going to access the light model
ILight abstractLight;
// We need to declare the two implems in order to get a reference
// to them from C++ code
MyModelLight1 modelLight1;
MyModelLight2 modelLight2;
float4 PixelShader(PS_INPUT Input ) : SV_Target
{
// Call the abstractLight that was previously setup by C++ at
// PixelShader creation time
float3 ambient = abstractLight.ComputeAmbient(Input.Pos);
float3 diffuse = abstractLight.ComputeDiffuse(Input.Pos);
float3 specular = abstractLight.ComputeSpecular(Input.Pos);
return float4(saturate( Ambient + Diffuse + Specular ), 1.0);
}
</pre>
<br />
To be able to use this shader, we need to setup the abstractLight variable from the C++/C# code, through the usage of <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476502%28v=vs.85%29.aspx">ID3D11Device::CreateClassLinkage</a> and in the instatiation of a Pixel Shader <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476513%28v=vs.85%29.aspx">ID3D11Device::CreatePixelShader</a>.<br />
<br />
As we can see, we need to declare the interface and classes variable globally, so that they can be accessed by the C++ program. This is the standard way to use dynamic linking in HLSL... but what If we want to use this differently?
<br />
<br />
<h3>
Hacking function pointers in HLSL</h3>
<br />
The principle is very simple: <b>Instead of using interface and classes as global variables, we can in fact use them as function parameters and even local variables from method</b>. The way to use it is then straightforward:
<br />
<pre class="brush: hlsl" name="code">// Base class for a calculator
interface ICalculator {
float Compute(...);
};
// 1st implem of the calculator
class ClassicCalculator : ICalculator {
float Compute(...) {
...
return value;
}
};
// 2nd implem of the calculator
class ComplexCalculator : ICalculator {
float Compute(...) {
...
return value;
}
};
// A function using the interface ICalculator
float MyFunctionUsingICalculator(ICalculator calculator, ...) {
...
value += calculator.Compute(...);
...
return value;
}
// A Pixel shader using the ClassicCalculator
float PixelShader1(PS_INPUT Input ) : SV_Target
{
ClassicCalculator classic;
return MyFunctionUsingICalculator(classic, ...);
}
// A Pixel shader using the ComplexCalculator
float PixelShader2(PS_INPUT Input ) : SV_Target
{
ComplexCalculator complex;
return MyFunctionUsingICalculator(complex, ...);
}
</pre>
<br />
The previous example could be compiled flawlessly with ps_4_0 (Shader Model 4) or ps_3_0 (with some minor changes for the pixel shader), and It would compile just fine!
So basically, <b>the interface ICalculator is acting as a function pointer</b>, that has two implementations available through the ClassicCalculator and ComplexCalculator classes. MyFunctionUsingICalculator doesn't have to change its signature to adapt to the underlying function, so as we can see, we have a suitable solution for developing function pointers in HLSL.
<br />
<br />
Now, lets try to see if we could use this model to build our flexible noise functions. Replace ICalculator by a INoise interface. We are seeing that an implementation would have to call another INoise interface. In fact, ideally, we would like to code something like this:
<br />
<pre class="brush: hlsl" name="code">// Base class for a noise function
interface INoise {
float Compute(...);
};
// Perlin noise implem
class PerlinNoise : INoise {
float Compute(...) {
...
return value;
}
};
// FBM noise implem
class FBMNoise : INoise {
// Would be ideal to be able to do that
// We could even make an abstract generic class
// that could provide a base Source INoise
// BUT, THIS IS NOT COMPILING!!!
INoise Source;
float Compute(...) {
float value = 0.0f;
float frequency = InitialFrequency;
float amplitude = 1.0f;
// Classic FBM loop
for ( int i=0; i < Octaves; i++ )
{
// Call the source abstract INoise
float noiseValue = Source.Compute(pos);
value += amplitude * noiseValue;
frequency *= Lacunarity;
amplitude *= Amplitude;
}
return value;
}
};
// A Pixel shader using the FBMNoise combined with PerlinNoise
float PixelShader1(PS_INPUT Input ) : SV_Target
{
FBMNoise fbmNoise;
PerlinNoise perlin;
// This is not possible, interface variable members are not allowed
fbmNoise.Source = perlin;
return fbmNoise.Compute(...);
}
</pre>
<br />
Unfortunately, <b>HLSL doesn't permit the use of interface as variable members!</b>. This limitation was quite annoying, as It excludes a whole range of combination, like aggregation, composition... making these function pointers useful only for a very limited set of cases...
<br />
I have tried to overcome this problem using abstract class instead of interface, as classes can be declared as variable members of classes... but, again, there is a huge limitation: The class variable is in fact acting a a final or const variable that cannot be changed, thus making its usage almost useless...
<br />
But I knew that HLSL permits lots of unusual constructions, and this is where closures are going to resolve this.
<br />
<br />
<h3>
Hacking Closures in HLSL</h3>
<br />
So we know that interfaces can be used as function pointers, but their usage is limited as we cannot use anykind of composition. An interesting fact is that we can declare local variables in methods as being class or interfaces... The trick is to use a quite uncommon feature of HLSL: <b>It is possible to declare local classes inside a method, that can access local parameters!</b>
Therefore, It is possible to use a kind of <b>deferred composition/aggregation using this technique</b>. Let's rewrite our noise functions using this new closure technique:
<br />
<br />
1. <b>Declare a INoise interface that is able to compute the noise by using a next INoise implementation</b>.
<br />
<br />
<pre class="brush: hlsl" name="code">// It is possible to compile this code under ps_4_0 and ps_3_0
// Declare our INoise interface
interface INoise {
// Here an interesting hack: We can declare a method that is returning a INoise
// interface. This method will be implemented by the pixel shaders.
INoise Next();
// The compute method of a Noise
float Compute(float2 pos);
};
</pre>
<br />
2. <b>Declare NoiseBase as an abstract implementation of INoise </b>that is implementing the methods. If we had the keyword abstract in hlsl we wouldn't have to implement methods of this class.
<br />
<br />
<pre class="brush: hlsl" name="code">// We are creating an abstract class from INoise in order
// to implement both methods
class NoiseBase : INoise {
INoise Next() {
// This code will never be used. It is only
// used to declare this class
NoiseBase base;
return base;
}
float Compute(float2 pos) {
// This code will never be used. It is only
// used to declare this class
return Next().Compute(pos);
}
};
</pre>
<br />
3. <b>Use NoiseBase to implement final INoise functions</b>. If you look at AbsNoise, FbmNoise or MarbleNoise, they are using the INoise::Next() method to get an instance of the INoise interface they rely on.
This is where functions pointers are extremely useful here.
<br />
<br />
<pre class="brush: hlsl" name="code">// PerlinNoise implem
class PerlinNoise : NoiseBase {
float Compute(float2 pos) {
// call a standard perlin_noise implemented as a simple external function
return perlin_noise(pos);
}
};
// AbsNoise implem
class AbsNoise : NoiseBase {
float Compute(float2 pos) {
// Note: We are using Next to access the next underlying function pointer
return abs(Next().Compute(pos));
}
};
// FbmNoise implem
class FbmNoise : NoiseBase {
float Compute(float2 pos) {
float value = 0.0f;
float amplitude = 1.0f;
float frequency = InitialFrequency;
for ( int i=0; i < Octaves; i++ )
{
float noiseValue = Next().Compute(pos);
value += amplitude * noiseValue;
frequency *= Lacunarity;
amplitude *= Amplitude;
}
return value;
}
};
// MarbleNoise implem
class MarbleNoise : NoiseBase {
float Compute(float2 pos) {
return stripes(2 * Next().Compute(pos, frequency), 1.6f);
}
static float stripes(float x, float f) {
float t = .5 + .5 * sin(f * 2*PI * x);
return t * t - .5;
}
};
</pre>
<br />
4. <b>Implements the pixel shaders with the closure mechanism</b>. We are declaring local classes that will override INoise::Next() method in order to chain INoise function pointers together.
<br />
<br />
<pre class="brush: hlsl" name="code">// Fbm -> PerlinNoise
float FbmPerlinNoise2DPS( float4 pos : SV_POSITION, float2 texPos : TEXCOORD0 )
: SV_Target
{
// Look! We are declaring a local class
class Noise1 : PerlinNoise {} noise1;
// and this local classs can access local variable!
// For example, Noise2 can access previous noise1 variable.
class Noise2 : FbmNoise { INoise Next() { return noise1; } } noise2;
// Allowing us to cascade the calls and making a kind of deferred composition.
return noise2.Compute(texPos);
}
// Fbm -> Abs -> PerlinNoise
float FbmAbsPerlinNoise2DPS( float4 pos : SV_POSITION, float2 texPos : TEXCOORD0 )
: SV_Target
{
class Noise1 : PerlinNoise {} noise1;
class Noise2 : AbsNoise { INoise Next() { return noise1; } } noise2;
class Noise3 : FbmNoise { INoise Next() { return noise2; } } noise3;
// FbmNoise is calling indirectly AbsNoise that will call PerlinNoise.
return noise3.Compute(texPos);
}
// Marble -> Fbm -> Abs -> PerlinNoise
float FbmAbsPerlinNoise2DPS( float4 pos : SV_POSITION, float2 texPos : TEXCOORD0 )
: SV_Target
{
class Noise1 : PerlinNoise {} noise1;
class Noise2 : AbsNoise { INoise Next() { return noise1; } } noise2;
class Noise3 : FbmNoise { INoise Next() { return noise2; } } noise3;
class Noise4 : MarbleNoise { INoise Next() { return noise3; } } noise4;
// MarbleNoise is calling FbmNoise that is calling indirectly AbsNoise
// that will call PerlinNoise.
return noise4.Compute(texPos);
}
// Fbm -> Marble -> Abs -> PerlinNoise
float FbmAbsPerlinNoise2DPS( float4 pos : SV_POSITION, float2 texPos : TEXCOORD0 )
: SV_Target
{
class Noise1 : PerlinNoise {} noise1;
class Noise2 : AbsNoise { INoise Next() { return noise1; } } noise2;
class Noise3 : MarbleNoise { INoise Next() { return noise2; } } noise3;
class Noise4 : FbmNoise { INoise Next() { return noise3; } } noise4;
// FbmNoise is calling MarbleNoise that is calling indirectly AbsNoise
// that will call PerlinNoise.
return noise4.Compute(texPos);
}
</pre>
Et voila! As you can see, we are able to declare local classes from a pixel shader that are acting as closures. It is for example even possible to declare local classes that have a specific code in their Compute() methods.<br />
Behind the scene, when chaining the INoise::Next() methods, the fxc HLSL compiler is seeing all thoses classes as "INoise*".<br />
It is then possible to perform a fbm(marble(abs(perlin_noise()))) as well as a marble(fbm(abs(perlin_noise()))). <br />
<br />
In the end, <b>It is effectively possible to implement closures in HLSL that can be used in SM4.0 as well as SM3.0</b>!<br />
<br />
<h3>
Improving closures chaining</h3>
<br />
From the previous example, we can extend the concept by<br />
1. Adding <b>static local constructors</b> to each Noise function :
<br />
<pre class="brush: hlsl" name="code">// PerlinNoise implem
class PerlinNoise : NoiseBase {
float Compute(float2 pos) {
// call a standard perlin_noise implemented as a simple external function
return perlin_noise(pos);
}
// Add local "constructor"
static INoise New() {
PerlinNoise noise;
return noise;
}
};
// AbsNoise implem
class AbsNoise : NoiseBase {
float Compute(float2 pos) {
// Note: We are using Next to access the next underlying function pointer
return abs(Next().Compute(pos));
}
// Add local constructor and chain with From INoise
static INoise New(INoise from) {
class LocalNoise : AbsNoise { INoise Next() { return from; } } noise;
return noise;
}
};
// Add the same constructors to FbmNoise and MarbleNoise.
// ....
</pre>
2. And then we can rewrite the Pixel shader functions to <b>chain operators in a shorter form</b>:
<br />
<pre class="brush: hlsl" name="code">// Fbm -> Marble -> Abs -> PerlinNoise
float FbmAbsPerlinNoise2DPS( float4 pos : SV_POSITION, float2 texPos : TEXCOORD0 )
: SV_Target
{
// FbmNoise is calling MarbleNoise that is calling indirectly AbsNoise
// that will call PerlinNoise.
return FbmNoise::New(MarbleNoise::New(AbsNoise::New(PerlinNoise::New()))).Compute(texPos);
}
</pre>
<br />
This way, It allows a syntax that is even more concise and modular!
<br />
<br />
<h3>
Further Considerations</h3>
<br />
This is a very exciting technique that could open lots of abstraction opportunities while developing in HLSL. Though, in order to use this technique, there are a couple of advantages and things to take into account:<br />
<ul>
<li><b>An interface cannot inherit from another interface </b>(that would be really interesting)</li>
<li><b>An interface can only have method members</b>.</li>
<li><b>A class can inherit from another class and from several interfaces</b>.</li>
<li>Unlike in C/C++, <b>we cannot pre-declare an interface</b>, but we can <b>use a declaration being declared </b>(See the example of the method INoise::Next, returning a INoise).</li>
<li><b>The compiler has a limitation against the reuse of an implementation in a call chain and will complain about a recursive call </b>(even if there is no recursive call at all): For example, It is not possible to reuse twice the sample type of class closure in a call chain, meaning that it is not possible to make a call chain like this one: Marble => FBM => Marble => Abs => Perlin. The fxc compiler would complain about the second "Marble" as It would see it as a kind of recursive call. <b>In order to reuse a function, we need to duplicate it</b>, that's probably the only point that is annoying here.</li>
<li><b>Generated compiled asm output from closures are exactly the same as using standard inlining </b>methods. </li>
<li>Before going to local class-closure, I have tried several techniques that were sometimes crashing fxc compiler.</li>
<li>Thus, as it is a way of hacking the usage HLSL, <b>It is not guarantee that this will be supported in the future</b>. But at least, if it is working for SM5.0, SM4.0 and 3.0, we can expect that we are safe for a while!</li>
<li>Also, the compilation time under vs_3_0/ps_3_0 profile seems to take more time, not sure if its the language construction or a regular behavior of 3.0 profiles.</li>
</ul>
Let me know if you are able to use this technique and If you are finding other interesting constructions or problems. That would be very interesting to dig a little more into the opportunities it opens. Lastly, I have done a small google search about this kind of technique, but didn't found anything... but It could have been used already by someone else, thus this whole technique is a new hypothetical discovery, but I enjoyed a lot to discover it!<br />
<script src="http://xoofx.com/syntaxhighlighter/highlighLoader.js" type='text/javascript'></script>xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com4tag:blogger.com,1999:blog-1076643699683521890.post-11102473599466409262011-11-18T00:46:00.001+11:002011-12-02T23:34:34.784+11:00Direct3D11 multithreading micro-benchmark in C# with SharpDX<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-mMXppMwXGFfUamNmZ2WmpjvJkrPv9366GIWvtsuikV5YU_CTXCEGmwMbGApsDJJUz0oaGfI3yEOsI2Jk7PiZdSJJ_8hPJPdaxyjKkQX8J9iswBVNGHuCDKUe-i1nvgYVe0FSboWKia0/s1600/MultiCube.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-mMXppMwXGFfUamNmZ2WmpjvJkrPv9366GIWvtsuikV5YU_CTXCEGmwMbGApsDJJUz0oaGfI3yEOsI2Jk7PiZdSJJ_8hPJPdaxyjKkQX8J9iswBVNGHuCDKUe-i1nvgYVe0FSboWKia0/s200/MultiCube.png" width="195" /></a></div>
Multi-threading is an important feature added to Direct3D11 two years ago and has been increasingly used on recent game engine in order to achieve better performance on PC. You can have a look at "<a href="http://www.slideshare.net/DICEStudio/directx-11-rendering-in-battlefield-3?from=ss_embed">DirectX 11 Rendering in Battlefield 3</a>" from Johan Anderson/DICE which gives a great insight about how it was effectively used in practice in their game engine. Usage of the Direct3D11 multithreading API is pretty straightforward, and while we are also using it successfully at our work in our R&D 3D Engine, I didn't take the time to sit down with this feature and check how to get the best of it.<br />
<br />
I recently came across a question on the gamedev forum about "<a href="http://www.gamedev.net/topic/614998-dx11-command-lists-on-a-single-threaded-renderer/">[DX11] Command Lists on a Single Threaded Renderer</a>": If command lists are an efficient way to store replayable drawing commands, would it be efficient to use them even in a single threaded scenario where lots of drawing commands are repeatable?<br />
<br />
In order to verify this, among other things, I did a simple micro-benchmark using <a href="http://code.google.com/p/sharpdx/">C#/SharpDX</a>, but while the results are somehow expectable, there are a couple of gotchas that deserve a more in-depth look...<br />
<br />
<a name='more'></a><br />
<h3>
Direct3D11 Multi-threading : The basics</h3>
<br />
I assume that general multi-threading concepts and advantages are already understood to focus on Direct3D11 multi-threading API.<br />
<br />
There is already a nice "<a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476891%28v=vs.85%29.aspx">Introduction to Multithreading in Direct3D11</a>" on msdn that is worth reading if you are already a little bit familiar with the Direct3D11 API.<br />
<br />
In Direct3D10, we had only a class <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/bb173528%28v=VS.85%29.aspx">ID3D10Device</a> to perform object/resource creation and draw calls, the API was not thread safe, but It was possible to emulate some kind of deferred rendering by using mutexes and a simplified command buffers to access safely the device.<br />
<br />
In Direct3D11, preparation of the draw calls are now "parralelizable" while object/resource creation is thread safe. The API is now split between:<br />
<ul>
<li><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476379%28v=VS.85%29.aspx">ID3D11Device</a> which is responsible to create object/resources/shaders and device contexts.</li>
<li><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476385%28v=VS.85%29.aspx">ID3D11DeviceContext</a> which holds all commands to setup shaders pipeline and perform all draw calls (including constant buffer update, setup of shader resource views, samplers, blendstate...etc.)</li>
</ul>
<br />
When a Direct3D11 device is created, it provides a default ID3D11DeviceContext called an <b>immediate context</b> that is effectively used for immediate rendering. There is only one immediate context available per device.<br />
<br />
In order to use deferred rendering, we need to create new ID3D11DeviceContext called <b>deferred context</b>. One context for each thread responsible for preparing a set of draw calls.<br />
<br />
Then the sequence of multithreaded draw calls are executed like this:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_CZH-OHGjmI4EzLuO0BlcH0QRtGONrXTKUOydJWWVN_jUVQbQNrE8bYAZ8dEec0OdS7ruK_uIqW0ebOIBXkGPkOWSObKr-vYboW-_VOzVMCy4hyphenhyphenwCVEXd9hw8k1XlKQ8XWUNkFfNfeNs/s1600/Direct3D11Multithreading.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="443" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_CZH-OHGjmI4EzLuO0BlcH0QRtGONrXTKUOydJWWVN_jUVQbQNrE8bYAZ8dEec0OdS7ruK_uIqW0ebOIBXkGPkOWSObKr-vYboW-_VOzVMCy4hyphenhyphenwCVEXd9hw8k1XlKQ8XWUNkFfNfeNs/s640/Direct3D11Multithreading.png" width="640" /></a></div>
Each secondary threads are responsible to prepare draw calls in a set of <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476361%28v=vs.85%29.aspx">ID3D11CommandList</a> that will effectively be executed by the immediate context (in order to push them to the driver).<br />
<br />
The simplified version of the code to write is fairly easy:<br />
<br />
<pre class="brush: csharp" name="code">// Thread-1
context[threadIdn].InputAssembler.InputLayout = layout1;
context[threadIdn].InputAssembler.PrimitiveTopology = PrimitiveTopology.TriangleList;
context[threadIdn].InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(vertices1, Utilities.SizeOf<Vector4>() * 2, 0));
[...]
context[threadId1].Draw(...)
commandLists[threadId1] = context[ThreadId1].FinishCommandList(false);
[...]
// Thread-n
context[threadIdn].InputAssembler.InputLayout = layoutn;
context[threadIdn].InputAssembler.PrimitiveTopology = PrimitiveTopology.TriangleList;
context[threadIdn].InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(verticesn, Utilities.SizeOf<Vector4>() * 2, 0));
[...]
context[threadIdn].Draw(...)
commandLists[threadIdn] = context[ThreadIdn].FinishCommandList(false);
// Rendering Thread
for (int i = 0; i < threadCount; i++)
{
var commandList = commandLists[i];
// Execute the deferred command list on the immediate context
immediateContext.ExecuteCommandList(commandList, false);
commandList.Dispose();
}
</pre>
<br />
The API provides several key advantages:<br />
<ul>
<li>We can <b>easily switch the code between immediate context and deferred context</b>. Thus using the multi-threading part of the Direct3D11 API doesn't hurt our code.</li>
<li>The <b>API is supported on downlevel hardware </b>(from Direct3D11 down to Direct3D9)</li>
<li>The underlying driver can take advantages when calling <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476424%28v=vs.85%29.aspx"><code>FinishCommandList</code></a> to perform some native layout that will help the deferred <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476423%28v=VS.85%29.aspx"><code>ExecuteCommandList</code></a> command to run faster.</li>
</ul>
About the "native support from driver", It can be checked by using <code>CheckFeatureSupport</code> (or directly in SharpDX using <code>CheckThreadingSupport</code>) but it seems that almost only NVIDIA (and quite recently, around this year), is supporting this feature natively. On my previous ATI 6850 and now on my 6900M are not supporting it. Is this bad? We will see that the default Direct3D11 runtime is performing just fine for this, but doesn't provide any extra boost.<br />
<br />
We will also see that there is an interesting issue with the usage of <code><a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476457%28v=VS.85%29.aspx">Map</a>/<a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476485%28v=VS.85%29.aspx">Unmap</a></code> or <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ff476486%28v=VS.85%29.aspx"><code>UpdateSubresource</code></a> in order to update constant buffers, and their respective usage under a multithreading scenario could hurt performances. <br />
<br />
<h3>
MultiCube, a Direct3D11 Multi-threading micro-benchmark</h3>
<br />
In order to stress-test multi-threading using Direct3D11, I have developed a simple application called MultiCube (available as part of SharpDX samples: See <a href="http://code.google.com/p/sharpdx/source/browse/Samples/Direct3D11/MultiCube/Program.cs">Program.cs</a>)<br />
<br />
<br />
This application is performing the following benchmark: It renders n x n cubes on the screen, each cube has its own matrix rotation. You can modify the number of cubes from 1 (1x1) to 65536 (256x256). The title bar is including some benchmark measurement (FPS/ time per frame) and you can change the behavior of the application with following keys:<br />
<ul>
<li><b>F1</b>: Switch between Immediate Test (no threading), Deferred Test (Threading), and Frozen-Deferred Test (execute a pre-prepared CommandList on the ImmediateContext)</li>
<li><b>F2</b>: Switch between Map/Unmap mode and UpdateSubresource mode to update constant buffers.</li>
<li><b>F3</b>: Burn the CPU on/off. This is were multithreading usage is making the difference and we are going to analyse the results a little bit more. When this option is on, It simulates lots of CPU calculation on the deferred threads. If this is off, It will just batch the draw calls (which are simple, its just Cubes!)</li>
<li><b>Left-Right arrows</b>: Decrease/Increase the number of cubes to display (default 64x64)</li>
<li><b>Down-Up arrows</b>: Decrease/Increase the number of threads used (only for Deferred Test mode)</li>
</ul>
When the deffered mode is selected, each threads are rendering a set of rows in batch. If you have for example 100x100 cubes to render, and 5 threads, each thread will draw 20x100 cubes.<br />
<br />
If your graphics driver doesn't support natively multithreading, you will see a "*" just after Deferred node.<br />
<br />
<b>You can download the application </b><a href="http://sharpdx.org/sample/SharpDX-MultiCube.zip">here</a>. It is a single exe that doesn't need anykind of install (apart the DirectX June 2010 runtime). Also, being able to pack this application into a single exe is a unique feature of SharpDX: <b>static linking of a .NET exe with SharpDX Dlls</b>.<br />
<br />
<br />
<h3>
Results</h3>
<br />
I ran 2 type of tests:<br />
<ol>
<li>Draw 65536 cubes with the Burn-Cpu option ON and OFF, and comparing Immediate and Deferred rendering (ranging from 1 thread to 6 threads). </li>
<li>Draw 1024 cubes switching between Map/Unmap and UpdateSubresource, and comparing the results between Immediate and Deferred rendering.</li>
</ol>
Two machines with the same main processor Intel i7-2600K, 8Go RAM were used, one with NVIDIA GTX 570 and the other one with a ATI 6900M graphics card.<br />
<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="margin: 0px auto; width: 558px;"><colgroup><col style="mso-width-alt: 8192; mso-width-source: userset; width: 168pt;" width="224"></col>
<col style="mso-width-alt: 2560; mso-width-source: userset; width: 53pt;" width="70"></col>
<col style="mso-width-alt: 2633; mso-width-source: userset; width: 54pt;" width="72"></col>
<col span="3" style="width: 48pt;" width="64"></col>
</colgroup><tbody>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt; width: 168pt;" width="224">65536 Drawcalls -
BurnCpu: On</td>
<td class="xl64" colspan="5" style="width: 251pt;" width="334">Threads</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">Type</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">3</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">6</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Deferred</td>
<td align="right" class="xl63" style="background: #F8696B; border: .5pt solid windowtext; mso-pattern: black none;">232ms</td>
<td align="right" class="xl63" style="background: #99CD7E; border: .5pt solid windowtext; mso-pattern: black none;">130ms</td>
<td align="right" class="xl63" style="background: #75C37C; border: .5pt solid windowtext; mso-pattern: black none;">98ms</td>
<td align="right" class="xl63" style="background: #6EC17B; border: .5pt solid windowtext; mso-pattern: black none;">92ms</td>
<td align="right" class="xl63" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">82ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Immediate</td>
<td align="right" class="xl63" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">220ms</td>
<td align="right" class="xl63" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">220ms</td>
<td align="right" class="xl63" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">220ms</td>
<td align="right" class="xl63" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">220ms</td>
<td align="right" class="xl63" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">220ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Deferred</td>
<td align="right" class="xl63" style="background: #F9746E; border: .5pt solid windowtext; mso-pattern: black none;">231ms</td>
<td align="right" class="xl63" style="background: #9ACD7E; border: .5pt solid windowtext; mso-pattern: black none;">131ms</td>
<td align="right" class="xl63" style="background: #75C37C; border: .5pt solid windowtext; mso-pattern: black none;">98ms</td>
<td align="right" class="xl63" style="background: #6FC17B; border: .5pt solid windowtext; mso-pattern: black none;">93ms</td>
<td align="right" class="xl63" style="background: #65BE7B; border: .5pt solid windowtext; mso-pattern: black none;">84ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Immediate</td>
<td align="right" class="xl63" style="background: #FB9574; border: .5pt solid windowtext; mso-pattern: black none;">228ms</td>
<td align="right" class="xl63" style="background: #FB9574; border: .5pt solid windowtext; mso-pattern: black none;">228ms</td>
<td align="right" class="xl63" style="background: #FB9574; border: .5pt solid windowtext; mso-pattern: black none;">228ms</td>
<td align="right" class="xl63" style="background: #FB9574; border: .5pt solid windowtext; mso-pattern: black none;">228ms</td>
<td align="right" class="xl63" style="background: #FB9574; border: .5pt solid windowtext; mso-pattern: black none;">228ms</td>
</tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvKAuXD94_NFUdxPIXSDAVV4bNT-Ytgp6nt4WB3vi9LGCAwclY8xwyAQ5EgU4LA-VUTloFfye3czigQ0Lnt6XPQPO1kymq3O0PvMXfkMVo6J0wp4uqGRiZR79j31lXhBusXq1iAdtnK2c/s1600/MultiCube-BurnCpuOn.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvKAuXD94_NFUdxPIXSDAVV4bNT-Ytgp6nt4WB3vi9LGCAwclY8xwyAQ5EgU4LA-VUTloFfye3czigQ0Lnt6XPQPO1kymq3O0PvMXfkMVo6J0wp4uqGRiZR79j31lXhBusXq1iAdtnK2c/s640/MultiCube-BurnCpuOn.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig2. 65536 draw calls with <b>CPU intensive </b>threads, comparison between Immediate and Deferred rendering</td></tr>
</tbody></table>
<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="margin: 0px auto; width: 558px;"><colgroup><col style="mso-width-alt: 8192; mso-width-source: userset; width: 168pt;" width="224"></col>
<col style="mso-width-alt: 2560; mso-width-source: userset; width: 53pt;" width="70"></col>
<col style="mso-width-alt: 2633; mso-width-source: userset; width: 54pt;" width="72"></col>
<col span="3" style="width: 48pt;" width="64"></col>
</colgroup><tbody>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="height: 15.0pt; width: 168pt;" width="224">65536 Drawcalls -
BurnCpu: Off</td>
<td class="xl66" colspan="5" style="width: 251pt;" width="334">Threads</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">Type</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">1</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">2</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">3</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">4</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">6</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Deferred</td>
<td align="right" class="xl65" style="background: #FA8A72; border: .5pt solid windowtext; mso-pattern: black none;">31ms</td>
<td align="right" class="xl65" style="background: #B9D780; border: .5pt solid windowtext; mso-pattern: black none;">24ms</td>
<td align="right" class="xl65" style="background: #85C87D; border: .5pt solid windowtext; mso-pattern: black none;">21ms</td>
<td align="right" class="xl65" style="background: #74C37C; border: .5pt solid windowtext; mso-pattern: black none;">20ms</td>
<td align="right" class="xl65" style="background: #74C37C; border: .5pt solid windowtext; mso-pattern: black none;">20ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Immediate</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">19ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">19ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">19ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">19ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">19ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Deferred</td>
<td align="right" class="xl65" style="background: #F8696B; border: .5pt solid windowtext; mso-pattern: black none;">32ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Immediate</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
<td align="right" class="xl65" style="background: #FFEB84; border: .5pt solid windowtext; mso-pattern: black none;">28ms</td>
</tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin: 0px auto;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyQmFL1neSFf9f6K28TwcEkanywGDgE85HHUezHreHO2JsCYe2woXUeM1yT__nKgdBFZxiqekP2mAkThdDkGMyKjscabOp8Q8Ls-KwdKk4Y4qVALvoYSD9EuD8i1bEL5D7U_uVIsQ8DFk/s1600/MultiCube-BurnCpuOff.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyQmFL1neSFf9f6K28TwcEkanywGDgE85HHUezHreHO2JsCYe2woXUeM1yT__nKgdBFZxiqekP2mAkThdDkGMyKjscabOp8Q8Ls-KwdKk4Y4qVALvoYSD9EuD8i1bEL5D7U_uVIsQ8DFk/s640/MultiCube-BurnCpuOff.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig2. 65536 draw calls with <b>CPU ligh </b>threads, comparison between Immediate and Deferred rendering</td></tr>
</tbody></table>
<br />
And finally the Map/Unmap and UpdateSubresource test:<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="width: 366px;"><colgroup><col style="mso-width-alt: 8192; mso-width-source: userset; width: 168pt;" width="224"></col>
<col style="mso-width-alt: 2560; mso-width-source: userset; width: 53pt;" width="70"></col>
<col style="mso-width-alt: 2633; mso-width-source: userset; width: 54pt;" width="72"></col>
</colgroup><tbody>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 168pt;" width="224">65536 Drawcalls - Type</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 53pt;" width="70">Map</td>
<td style="background: #1F497D; border: .5pt solid windowtext; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 54pt;" width="72">Update</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Immediate - 1024</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">0.6ms</td>
<td align="right" class="xl65" style="background: #FFE283; border: .5pt solid windowtext; mso-pattern: black none;">1.1ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">Nvidia-GTX
570 Deferred - 1024</td>
<td align="right" class="xl65" style="background: #FFE583; border: .5pt solid windowtext; mso-pattern: black none;">0.92ms</td>
<td align="right" class="xl65" style="background: #F8696B; border: .5pt solid windowtext; mso-pattern: black none;">7.32ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Immediate - 1024</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">0.6ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">0.6ms</td>
</tr>
<tr height="20" style="height: 15.0pt;">
<td height="20" style="border: .5pt solid windowtext; height: 15.0pt;">ATI 6900M
Deferred - 1024</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">0.6ms</td>
<td align="right" class="xl65" style="background: #63BE7B; border: .5pt solid windowtext; mso-pattern: black none;">0.6ms</td>
</tr>
</tbody></table>
<br />
<br />
<h3>
Analysis</h3>
<br />
If we examine the results a little more carefully, there are a couple of interesting things to highlight:<br />
<br />
<ul>
<li> <b>Using multithreading and deferred context rendering is only relevant when the CPU is effectively used on each threads </b>(that sounds obvious, but It is at least clear!). When we are not using the CPU, Immediate rendering is in fact faster!</li>
<li><b>Multithreading rendering with CPU intensive application can perform 3-4x times faster </b>than a single threaded application (at the condition that we have enough CPU core to dispatch rendering jobs)</li>
<li><b>The "native support from driver" of Direct3D11 multithreading doesn't seem to change so much</b>, compare to the NVIDIA graphics card that is supporting it, we don't see a huge difference with AMD.</li>
<li><b>Usage of UpdateSubresource on a NVIDIA card is 8x times slower in a multithreading situation </b>and is hurting a lot the performance of the application: <b>Use Map/Unmap instead</b>! </li>
</ul>
Of course, as usual, this is a synthetic, micro-benchmark test that should be taken with caution and can not reflect every test cases, so you need to perform your own benchmark if you have to make the decision of using multithreading rendering!<br />
<br />
Finally, to respond to the original gamedev question, I provided a "Frozen Deferred" test in MultiCube to test if rendering a pre-prepared CommandList is actually faster then executing it with an immediate context: It seems that It doesn't make currently any differences (but for this to be sure, I would have to run this benchmark on several different machines/CPU/graphics card/drivers configs in order to fully verify it).xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com5tag:blogger.com,1999:blog-1076643699683521890.post-22049296893615551472011-10-02T21:19:00.002+11:002011-10-06T23:37:11.103+11:00Managed DirectX with Win8 Metro Style App<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH11yHULobAVFr4uXyN7DC8n9UKyJIrwGUo1ln5LOE4UWBtGpt4BVWCSx8dM4rJDA-kEwzyjyiSd7p-mxKFkAOneQhB9exRPGd0s-x0pd1Qq4dgt2FrJCiX9eeXxm6svvE0FSEWd-iISs/s1600/logo_100x100.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH11yHULobAVFr4uXyN7DC8n9UKyJIrwGUo1ln5LOE4UWBtGpt4BVWCSx8dM4rJDA-kEwzyjyiSd7p-mxKFkAOneQhB9exRPGd0s-x0pd1Qq4dgt2FrJCiX9eeXxm6svvE0FSEWd-iISs/s1600/logo_100x100.png" /></a></div>
<span class="" id="result_box" lang="en"><span class="hps">Since my last post is quite old, it's time to give more insight about the latest features included in the recent release of</span><span class="hps"> <a href="http://code.google.com/p/sharpdx/">SharpDX 2.0</a> beta. This release is a major step for SharpDX as it includes lots of new APIs but also provides a preview of developing Managed DirectX from a Windows 8 Metro style application. In the meantime, I have been pretty busy as I got a job in a gamedev company located in Japan that is developing a new rendering engine in C#... which is using SharpDX for its windows rendering, and that's extremely exciting to work on a project like this (and of course rewarding for all the investement done in SharpDX!).</span></span><br />
<br />
<span class="" id="result_box" lang="en"><span class="hps">While SharpDX is starting to cover almost the whole DirectX API as well as some new Windows Multimedia API (like WIC), I put some effort, just after the Microsoft Build conference, on providing a preview of using SharpDX from a Win8 Metro style application, which is a fantastic opportunity for a C#/.Net developer to use DirectX from both a Metro style application and Desktop application using the same API.</span></span><br />
<br />
<span class="" id="result_box" lang="en"><span class="hps">Finally SharpDX is also going to get some new APIs and some interesting features in the following months, especially in the performance domain, to reduce as much as possible the difference of performance between a native C++ DirectX application and a managed C# DirectX application, that will deserve a post on its own. But lets start with what has been achieved in the latest 2.0 version!</span></span><br />
<a name='more'></a><br />
<h3>
SharpGen a new C# code generation tools from C++ </h3>
<br />
The core changes behind the 2.0 version was the rewrite of the code generator used to generate C# interop code from DirectX/Windows SDK headers, a tool called <b>SharpGen</b>. When I released the <a href="http://code4k.blogspot.com/2010/11/official-release-of-sharpdx-10.html">first version</a> of SharpDX, I used a handwritten C++ custom parser that was somewhat able to parse almost all DirectX headers, but It was of course a temporary workaround... End of December last year, I was evaluating two options to <b>rewrite the parser stage</b>:<br />
<ul>
<li><a href="http://clang.llvm.org/">CLang</a> seemed to be a suitable solution for the future, but at that time, I was not sure that It would be able to parse all windows headers flawlessly (while being able to grab all Microsoft specialized SAL-annotations from their headers in order to extract useful information used by the parser) and It was not really easy to handle (I need to wrap the library into a managed component).</li>
<li><a href="http://gccxml.org/HTML/Index.html">gccxml</a> is an older project, but is able to parse almost all Microsoft headers with a direct command line and It is quite easy to patch the headers that are not working. Gccxml is generating an xml file, which is pretty easy to parse.</li>
</ul>
So I made the choice to go with gccxml. Bur this is fortunately not a huge design issue, as it is quite easy in the current system to plug another parser, because the C++ model used by SharpGen is independent from the parser.<br />
<br />
The parser was one thing to rewrite in the old system. <b>The other part was to use a new data-driven config files for all mapping rules</b> (used to translate C++ objects to C#). The previous version was quickly developed using hard-coded rules directly written in the C# program. The new version is using simple xml config files to express all the mappings and dependencies. The good thing is that I didn't have to change so many things in the code generator, though lots of work was required to efficiently manage those configs files and their dependencies. You can have a look at the result, for example with <a href="http://code.google.com/p/sharpdx/source/browse/Source/SharpDX.DXGI/Mapping.xml">SharpDX.DXGI mapping file</a>. The new system has lots of features to manage all the cases that were found during the construction of all these mapping rules. It is also able to efficiently generate a subset of the files, when only parts of the config files are changed and they don't affect other dependent projects (For example, SharpDX.Direct3D11 has some dependencies on D3DCompiler, DXGI and the core SharpDX assembly. If any of the dependent config files are changed, I had to regenerate Direct3D11 as well). All this hard work was clearly done in order to have a system easier to maintain and to fix. The new generator is also able to handle all Windows headers files, including all multimedia files that were not part of the DirectX SDK headers (like WIC).<br />
<br />
Thanks to this new tool, It is now possible to integrate in the generated SharpDX assemblies some APIs that were not part of DirectX, but are hightly related to the development of multimedia applications, and that's really a good news for any developer seeking for an unified managed API for all the Windows Graphics and Multimedia APIs.<br />
<br />
I do plan also to release SharpGen as an external tool so that you can use it in your own project. It will allow a developer to generate easily interop mapping from C++ headers (with a COM oriented APIs)<br />
<br />
<br />
<h3>
Mapping all DirectX APIs </h3>
<br />
The next step was to write mapping rules and C# extensions for all the remaining APIs. The first version of SharpDX was providing managed APIs for DXGI, Direct3D10, Direct3D11, Direct2D, DirectWrite, DirectSound, XAudio2 but I had to work on older APIs that were more laborious to map like Direct3D9, DirectInput or minor APIs like X3DAudio, XACT3, RawInput, XInput.<br />
<br />
I decided also to provide a managed API for WIC (Windows Imaging Component) that is tightly used with Direct2D.<br />
<br />
So far, the result is great, except for Direct3D9 that still requires more work, all the DirectX APIs are now provided. SharpDX is also the only managed API to provide an implementation for all the callbacks used in Direct2D/DirectWrite, allowing a full access of these APIs.<br />
<br />
I took also the opportunity to rewrite the callback system (C++ is calling back C# objects as they are exposed to the C++ as COM objects). The new system is more efficient and reliable.<br />
<br />
<h3>
Using SharpDX from a Win8 Metro Style Application</h3>
<br />
A new exciting feature that was really easy to provide is the ability to <b>use SharpDX from a Win8 Metro Style Application</b>. In the meantime, <b>the 2.0 version is also providing an access to the upcoming DirectX11.1 APIs</b> (that includes DXGI 1.2, Direct2D.1 Direct3D11.1, new D3DCompiler, new XAudio2... etc.).<br />
<br />
Two samples were ported: one from the Win8 DirectX tutorials that is only clearing the screen. The other is a direct port of SharpDX Direct3D11 <a href="http://code.google.com/p/sharpdx/source/browse/Samples/Direct3D11/MiniCube/Program.cs">MiniCube</a>, a simple application that displays a 3D spinning cube on the screen. You can download the preview archive, compile and run it from a Win8 machine!<br />
<br />
In order to be fully compatible with Win8 metro application and to be able to develop application that will get certification on the Microsoft AppStore, some SharpDX internals are being rewritten in order to remove dependencies to legacy Win32 and .Net Framework APIs that are not supported by metro style application. <b>Next version of SharpDX will provide assemblies that will be fully compatible and certified-ready for Win8 Metro style application</b>.<br />
<br />
The great thing is that It will be possible to target desktop application and Win8 metro style application using the same managed DirectX API.<br />
<br />
You may ask if pure DirectX WinRT components could be generated using SharpDX technology? : in short, yes It is possible (as SharpGen could be modified to generate interop code for other languages), but I don't think this a a good opportunity for .NET developers, as the WinRT system will only be available to pure Win8 Metro style Application. Plus, WinRT has a significant performance cost when you want to have access to static properties/methods (that implies severals method calls : QueryInterface..etc).<br />
<br />
Also, I'm not entirely convinced that the new WinRT COM interop projection in .NET is as fast as custom interop used by SharpDX. I didn't have yet the time to write a full benchmark test, but I found some unnecessary generated interop methods by using WinRT components from .NET. So when I will have a chance to install Win8 on a real machine (and not in the VM), I will post a more detailed benchmark about WinRT calling cost from a managed application. <br />
<br />
<br />
<h3>
Next?</h3>
<br />
There are lots of new exciting features that will also be available in next versions of SharpDX, but I can't talk about them yet. In short, SharpDX will get new APIs binding for some Windows multimedia APIs (for example, someone suggested me on twitter to provide the <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd371981%28v=VS.85%29.aspx">UIAnimation Manager API </a>that I just started today) and will provide some great performance benefits as well.<br />
<br />
<br />
Also, I have seen a growing interest from XNA developers about SharpDX. It seems that the lack of communication about the future of XNA during and after the Build conference has generated lots of trouble in the XNA community. I don't know if XNA is going to be abandoned by Microsoft. It may be probably a bit to early to anticipate this. On the other hand, I wrote a couple of months ago a significant part of the Graphics layer of XNA on top of SharpDX, using internally Direct3D11, but I didn't release it, as it was not yet in a usable state. After the final 2.0 version of SharpDX will be released, I will have a look to see if this framework could be released as-is, in a state that would require some work from the SharpDX community.<br />
<br />
Finally, I'm receiving some great feedbacks from developers using SharpDX in some commercial projects and that's really great! I would be glad to hear more news from users, even If it can't be public, that's extremely interesting to understand SharpDX usages and improve the API experience as well.<br />
<br />
Stay tuned!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com9tag:blogger.com,1999:blog-1076643699683521890.post-47351166591808525032011-03-15T00:11:00.004+11:002014-06-18T01:04:09.402+11:00Benchmarking C#/.Net Direct3D 11 APIs vs native C++<div class="separator" style="clear: both; text-align: center;">
</div>
<span style="font-size: x-small;">[<b>Update 2012/05/15</b>: Note that the original code was fine tuned to a particular config and may not give you the same results. I have rewritten this sample to give more accurate and predictible results. Comparison with XNA was also not fair and inaccurate, you should have something like x4 slower instead of x9. SharpDX latest version 2.1.0 is also x1.35 slower than C++ now. An update of this article will follow on new <a href="http://sharpdx.org/">sharpdx.org</a> website]</span><br />
<span style="font-size: x-small;">[<b>Update 2014/06/17</b>: Remove XNA comparison, as it is not fair and relevant] </span><br />
<br />
If you are working with a managed language like C# and you are concerned by performance, you probably know that, even if the Microsoft JIT CLR is quite efficient, It has a significant cost over a pure C++ implementation. If you don't know much about this cost, you have probably heard about a mean cost for managed languages around 15-20%. If you are really concern by this, and depending on the cases, you know that the reality of a calculation-intensive managed application is more often around x2 or even x3 slower than its C++ counterpart. In this post, I'm going to present a micro-benchmark that measure the cost of calling a native Direct3D 11 API from a C# application, using various API, ranging from <a href="http://code.google.com/p/sharpdx/">SharpDX</a>, SlimDX, WindowsCodecPack.<br />
<br />
Why this benchmark is important? Well, if you intend like me to build some serious 3D games with a C# managed API (don't troll me on this! ;) ), you need to know exactly what is the cost of calling intensively a native Direct3D API (mainly, the cost of the interop between a managed language and a native API) from a managed language. If your game is GPU bounded, you are unlikely to see any differences here. But if you want to apply lots of effects, with various models, particles, materials, playing with several rendering targets and a heavy deferred rendering technique, you are likely to perform lots of draw calls to the Direct3D API. For a AAA game, those calls could be as high as 3000-7000 draw submissions in instancing scenarios (look at <a href="http://publications.dice.se/">latest great DICE publications </a>in "DirectX 11 Rendering in Battlefield 3" from Johan Andersson). If you are running at 60fps (or lower 30fps), you just have 17ms (or 34ms) per frame to perform your whole rendering. In this short time range, drawing calls can take a significant amount of time, and this is a main reason why multi-threading batching command were introduced in DirectX11. We won't use such a technique here, as we want to evaluate raw calls.<br />
<br />
As you are going to see, results are pretty interesting for someone that is concerned by performance and writing C# games (or even efficient tools for a 3D Middleware)<br />
<br />
<a name='more'></a><br />
<h3>
The Managed (C#) to Native (C++) interop cost</h3>
<br />
When a managed application needs to call a native API, it needs to:<br />
<ul>
<li>Marshal method/function arguments from the managed world to the unmanaged world</li>
<li>The CLR has to switch from a managed execution to an unmanaged environment (change exception handling, stacktrace state...etc.)</li>
<li>The native methods is effectively called</li>
<li>Than you have to marshal output arguments and results from unmanaged world to managed one.</li>
</ul>
To perform a native call from a managed language, there is currently 3 solutions: <br />
<ul>
<li><b>Using the default interop mechanism provided under C# is P/Invoke</b>, which is in charge of performing all the previous steps. But P/Invoke comes at a huge cost when you have to pass some structures, arrays by values, strings...etc. </li>
<li><b>Using a C++/CLI </b>assembly that will perform a marshaling written by hand to the native C++ methods. This is used by SlimDX, WindowsCodePack and XNA.</li>
<li><b>Using SharpDX technique </b>that is generating all the marshaling and interop at compile time, in a structured and consistent way, using some missing CLR bytecode inside C# that is usually only available in C++/CLI</li>
</ul>
The marshal cost is in fact the most expensive one. Usually, calling directly a native function without performing any marshaling has a cost of 10% which is fine. But if you take for example a slightly more complex functions, like <a href="http://msdn.microsoft.com/en-us/library/ff476464%28v=vs.85%29.aspx"><code>ID3D11DeviceContext::SetRenderTargets</code></a>, you can see that marshaling takes a significant amount of code:<br />
<br />
<pre class="brush: csharp" name="code">/// <unmanaged>void ID3D11DeviceContext::OMSetRenderTargets([In] int NumViews,[In, Buffer, Optional] const ID3D11RenderTargetView** ppRenderTargetViews,[In, Optional] ID3D11DepthStencilView* pDepthStencilView)</unmanaged>
public void SetRenderTargets(int numViews, SharpDX.Direct3D11.RenderTargetView[] renderTargetViewsRef, SharpDX.Direct3D11.DepthStencilView depthStencilViewRef) {
unsafe {
IntPtr* renderTargetViewsRef_ = (IntPtr*)0;
if ( renderTargetViewsRef != null ) {
IntPtr* renderTargetViewsRef__ = stackalloc IntPtr[renderTargetViewsRef.Length];
renderTargetViewsRef_ = renderTargetViewsRef__;
for (int i = 0; i < renderTargetViewsRef.Length; i++)
renderTargetViewsRef_[i] = (renderTargetViewsRef[i] == null)? IntPtr.Zero : renderTargetViewsRef[i].NativePointer;
}
SharpDX.Direct3D11.LocalInterop.Callivoid(_nativePointer, numViews, renderTargetViewsRef_, (void*)((depthStencilViewRef == null)?IntPtr.Zero:depthStencilViewRef.NativePointer),((void**)(*(void**)_nativePointer))[33]);
}
}
</pre>
In the previous sample, there is no structure marshaling involved (that are even more costly than pure method arguments marshaling), and as you can see, the marshaling code is pretty heavy: It has to handles null parameters, transform an array of managed DirectX interfaces to a respective array of native COM pointer...etc.<br />
<br />
Hopefully, in SharpDX unlike any other DirectX .NET APIs, this code has been written to be consistent over the whole generated code, and was carefully designed to be quite efficient... but still, It has obviously a cost, and we need to know it!<br />
<br />
<h3>
Protocol used for this micro-benchmark</h3>
<br />
Writing a benchmark is error prone, often subject to caution and relatively "narrow minded". Of course, this benchmark is not perfect, I just hope that It doesn't contain any mistake that would give false results trend!<br />
<br />
In order for this test to be closer to a real 3D application usage, I made the choice to perform a very basic test on a sequence of draw calls that are usually involved in common drawing calls scenarios. This test consist of drawing triangles using 10 successive effects (Vertex Shaders/Pixel Shaders), with their own vertex buffers, setting the viewport and render target to the backbuffer. This loop is then ran thousand of times in order to get a correct average.<br />
<br />
The SharpDX main loop is coded like this:<br />
<pre class="brush: csharp" name="code">var clock = new Stopwatch();
clock.Start();
for (int j = 0; j < (CommonBench.NbTests + 1); j++)
{
for (int i = 0; i < CommonBench.NbEffects; i++)
{
context.InputAssembler.SetInputLayout(layout);
context.InputAssembler.SetPrimitiveTopology(PrimitiveTopology.TriangleList);
context.InputAssembler.SetVertexBuffers(0, vertexBufferBindings[i]);
context.VertexShader.Set(vertexShaders[i]);
context.Rasterizer.SetViewports(viewPort);
context.PixelShader.Set(pixelShaders[i]);
context.OutputMerger.SetTargets(renderView);
context.ClearRenderTargetView(renderView, blackColor);
context.Draw(3, 0);
}
if (j > 0 && (j % CommonBench.FlushLimit) == 0)
{
clock.Stop();
Console.Write("{0} ({3}) - Time per pass {1:0.000000}ms - {2:000}%\r", programName, (double)clock.ElapsedMilliseconds / (j * CommonBench.NbEffects), j * 100 / (CommonBench.NbTests), arch);
context.Flush();
clock.Start();
}
}
</pre>
The VertexShader/PixelShaders involved is basic (just color passing between VS and PS, no WorldProjectionTransform applied), the context.Flush is used to avoid measuring flush of commands to the GPU. The <code>CommonBench.FlushLimit</code> value was selected to avoid any stalls from the GPU.<br />
<br />
I have ported this benchmark under:<br />
<ul>
<li><b>C++</b>, using raw native calls and Direct3D11 API</li>
<li><b>SharpDX</b>, using Direct3D11 running under Microsoft .NET CLR 4.0 and with Mono 2.10 (both trying llvm on/off). SharpDX is the only managed API to be able to run under Mono.</li>
<li><b>SlimDX </b>using Direct3D11 running under Microsoft .NET CLR 4.0. SlimDX is "NGENed" meaning that it is compiled to native code when you install it.</li>
<li><b>WindowsCodePack </b>1.1 using Direct3D11 running under Microsoft .NET CLR 4.0</li>
</ul>
It has been tested on a Win7-64bit, i5-750 2.6Ghz, Gfx AMD HD6950. All tests were done both in x86 and x64 mode, in order to measure the platform impact of the calling conventions. Tests were ran 4 times for each API, taking the average of the 3 lowest one.<br />
<br />
<h3>
Results</h3>
<br />
You can see the raw results in the following table. Time is measured for the simple drawing sequence (inside the loop for(i) nbEffects). Lower is better. The ratio on the right indicates how much is slower the tested API compare to the C++ one. For example, SharpDX in x86 mode is running 1,52 slower than its pure C++ counterpart.<br />
<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="width: 541px;"><colgroup><col style="mso-width-alt: 8740; mso-width-source: userset; width: 179pt;" width="239"></col> <col span="2" style="mso-width-alt: 2706; mso-width-source: userset; width: 56pt;" width="74"></col> <col span="2" style="mso-width-alt: 2816; mso-width-source: userset; width: 58pt;" width="77"></col> </colgroup><tbody>
<tr height="20" style="height: 15.0pt;"> <td class="xl63" height="20" style="background: #4F81BD; border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 700; height: 15.0pt; mso-pattern: #4F81BD none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 179pt;" width="239">Direct3D11 Simple Bench</td> <td class="xl63" style="background: #4F81BD; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 700; mso-pattern: #4F81BD none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 56pt;" width="74">x86 (ms)</td> <td class="xl63" style="background: #4F81BD; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 700; mso-pattern: #4F81BD none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 56pt;" width="74">x64 (ms)</td> <td class="xl63" style="background: #4F81BD; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 700; mso-pattern: #4F81BD none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 58pt;" width="77">x86-ratio</td> <td class="xl63" style="background: #4F81BD; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: white; font-family: Calibri; font-size: 11.0pt; font-weight: 700; mso-pattern: #4F81BD none; text-decoration: none; text-line-through: none; text-underline-style: none; width: 58pt;" width="77">x64-ratio</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;">Native C++ (MSVC VS2010)</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000386</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000262</td> <td class="xl64" style="background: #63BE7B; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x1.00</td> <td class="xl64" style="background: #63BE7B; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x1.00</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;">Managed SharpDX (1.3 MS .Net CLR)</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000585</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000607</td> <td class="xl64" style="background: #84C77C; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x1.52</td> <td class="xl64" style="background: #B9D67F; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x2.32</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;">Managed SlimDX (June 2010 - Ngen)</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000945</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.000886</td> <td class="xl64" style="background: #C1D980; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x2.45</td> <td class="xl64" style="background: #FFEB84; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x3.38</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;">Managed SharpDX (1.3 Mono-2.10)</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.002404</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.001872</td> <td class="xl64" style="background: #FDC27D; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x6.23</td> <td class="xl64" style="background: #FDB57A; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x7.15</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: none; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;">Managed Windows API CodePack 1.1</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.002551</td> <td class="xl63" style="border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;">0.003219</td> <td class="xl64" style="background: #FDBC7B; border-bottom: none; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x6.61</td> <td class="xl64" style="background: #F8696B; border-bottom: none; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;">x12.29</td> </tr>
<tr height="20" style="height: 15.0pt;"> <td height="20" style="border-bottom: .5pt solid #4F81BD; border-left: .5pt solid #4F81BD; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; height: 15.0pt; text-decoration: none; text-line-through: none; text-underline-style: none;"><br /></td> <td class="xl63" style="border-bottom: .5pt solid #4F81BD; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;"><br /></td> <td class="xl63" style="border-bottom: .5pt solid #4F81BD; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;"><br /></td> <td class="xl64" style="background: #FB8F73; border-bottom: .5pt solid #4F81BD; border-left: none; border-right: none; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; mso-pattern: black none; text-decoration: none; text-line-through: none; text-underline-style: none;"><br /></td> <td class="xl64" style="border-bottom: .5pt solid #4F81BD; border-left: none; border-right: .5pt solid #4F81BD; border-top: .5pt solid #4F81BD; color: black; font-family: Calibri; font-size: 11.0pt; font-weight: 400; text-decoration: none; text-line-through: none; text-underline-style: none;"><br /></td> </tr>
</tbody></table>
<br />
And the associated graphs comparison both for x86 and x64 platforms:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheW7aeCvpbNL_dzljM9j6QIhJkSpf6UyNbQgN6PrWlmvxu3Tf8XRW0O2NyUqJFoCxxOYQNBOSFODmG4Cin1Uq4djoWphfBQMDpN4yeSqlr4rvM4VU0eL4AeWOh58J4XguI-_boC-bzedA/s1600/Resultsx86.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheW7aeCvpbNL_dzljM9j6QIhJkSpf6UyNbQgN6PrWlmvxu3Tf8XRW0O2NyUqJFoCxxOYQNBOSFODmG4Cin1Uq4djoWphfBQMDpN4yeSqlr4rvM4VU0eL4AeWOh58J4XguI-_boC-bzedA/s1600/Resultsx86.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl0E6iyjjiolEg5L52eLLR2oPmkmlp8pAl0D1Jo4EBxbTxCQDH0DYwfI2IDIXvTaYnU9u0E04edmidPPC8LurULv34JcFN-qWIn1hWkiaA4nF4hhrHkACa-WnuV9ZjPILdsdVJmPCd_0w/s1600/Resultsx64.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhl0E6iyjjiolEg5L52eLLR2oPmkmlp8pAl0D1Jo4EBxbTxCQDH0DYwfI2IDIXvTaYnU9u0E04edmidPPC8LurULv34JcFN-qWIn1hWkiaA4nF4hhrHkACa-WnuV9ZjPILdsdVJmPCd_0w/s1600/Resultsx64.png" /></a></div>
<br />
Results are pretty self explanatory. Although we can highlight some interesting facts:<br />
<ul>
<li><b>Managed Direct3D API calls are much slower than native API calls</b>, ranging from x1.52 to x10 depending on the API you are using.</li>
<li><b>SharpDX is providing the fastest Direct3D managed API</b>, which is ranging only from x1.52 to x2.32 slower than C++, at least 50% faster than any other managed APIs.</li>
<li><b>All other Direct3D managed API are significantly slower</b>, ranging from x2.45 to x12.29</li>
<li><b>Running this benchmark with SharpDX and Mono 2.10 is x6 to x7 times slower than SharpDX with Microsoft JIT </b>(!)</li>
</ul>
Ok, so if you are a .NET programmer and are not aware about performance penalty using a managed language, you are probably surprised by these results that could be... scary! Although, we can balance things here, as your 3D engine is unlikely to be CPU bounded on drawing calls, but 3000-7000 calls could lead to a 4ms impact in the better case, which is something we need to know when we design a game.<br />
<br />
This test could be also extrapolated to other parts of a 3D engine, as It will probably slower by a factor of x2 compare to a brute force C++ engine. For AAA game, this would be of course an unacceptable performance penalty, but If you are a small/independent studio, this cost is relatively low compare to the cost of efficiently developing a game in C#, and in the end, that's a trade-off.<br />
<br />
In case you are using SharpDX API, you can still run at a reasonable performance. And if you really want to circumvent this interop cost for chatty API scenarios, you can design your engine to call a native function that will batch calls to the Direct3D native API. <br />
<br />
<hr />
You can download this benchmark <a href="http://xoofx.com/upload/Sharp3DBench.7z">Sharp3DBench.7z</a>.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com47tag:blogger.com,1999:blog-1076643699683521890.post-41886313471400307382010-12-29T04:41:00.027+11:002011-01-06T02:41:04.780+11:00Crinkler secrets, 4k intro executable compressor at its best<span style="font-size: x-small;">(Edit 5 Jan 2011: New <a href="http://code4k.blogspot.com/2010/12/crinkler-secrets-4k-intro-executable.html#results">Compression results</a> section and small crinkler x86 decompressor analysis)</span><br />
<br />
If you are not familiar with 4k intros, you may wonder how things are organized at the executable level to achieve this kind of packing-performance. Probably the most important and essential aspect of 4k-64k intros is the compressor, and surprisingly, 4k intros have been well equipped for the past five years, as <a href="http://crinkler.net/">Crinkler </a>is the best compressor developed so far for this category. It has been created by Blueberry (Loonies) and Mentor (tbc), two of the greatest demomakers around.<br />
<br />
Last year, I started to learn a bit more about the compression technique used in Crinkler. It started from some pouet's comments that intrigued me, like "crinkler needs several hundred of mega-bytes to compress/decompress a 4k intros" (wow) or "when you want to compress an executable, It can take hours, depending on the compressor parameters"... I observed also <a href="http://www.pouet.net/topic.php?which=6408&page=1&x=29&y=6">bad comrpession result</a>, while trying to convert some part of C++ code to asm code using crinkler... With this silly question, I realized that in order to achieve better compression ratio, you better need a code that is comrpession friendly but is not necessarily smaller. Or in other term, the smaller asm code is not always the best candidate for better compression under crinkler... so right, I needed to understand how crinkler was working in order to code crinkler-friendly code...<br />
<br />
I just had a basic knowledge about compression, probably the last book I bought about compression was more than 15 years ago to make a presentation about jpeg compression for a physics courses (that was a way to talk about computer related things in a non-computer course!)... I remember that I didn't go further in the book, and stopped just before arithmetic encoding. Too bad, that's exactly one part of crinkler's compression technique, and has been widely used for the past few years (and studied for the past 40 years!), especially in compressors like H.264!<br />
<br />
So wow, It took me a substantial amount of time to jump again on the compressor's train and to read all those complicated-statistical articles to understand how things are working... but that was worth it! In the same time, I spent a bit of my time to dissect crinkler's decompressor, extract the code decompressor in order to comment it and to compare its implementation with my little-own-test in this field... I had a great time to do this, although, in the end, I found that whatever I could do, under 4k, Crinkler is probably the best compressor ever.<br />
<br />
You will find here an attempt to explain a little bit more what's behind Crinkler. I'm far from being a compressor expert, so if you are familiar with context-modeling, this post may sounds a bit light, but I'm sure It could be of some interest for people like me, that are discovering things like this and want to understand how they make 4k intros possible!<br />
<br />
<a name='more'></a><br />
<h2>Crinkler main principles</h2><br />
If you want a bit more information, you should have a look at the "manual.txt" file in the crinkler's archive. You will find here lots of valuable information ranging from why this project was created to what kind of options you can setup for crinkler. There is also an old but still accurate and worth to look at powerpoint presentation from the author themselves that is available <a href="ftp://ftp.scene.org/pub/parties/2005/assembly05/seminars/crinkler-compression.ppt">here</a>.<br />
<br />
First of all, you will find that crinkler is not strictly speaking an executable compressor but is rather <b>an integrated linker-compressor</b>. In fact, in the intro dev tool chain, It's used as part of the building process and is used inplace of your traditional linker.... while crinkler has the ability to compress its output. Why crinkler is better suited at this place? Most notably because at the linker level, crinkler has access to portions of your code, your data, and is able to move them around in order to achieve better compression. Though, for this choice, I'm not completely sure, but this could be also implemented as a standard exe compressor, relying on relocation tables in the PE sections of the executable and a good disassembler like <a href="http://www.beaengine.org/">beaengine</a> in order to move the code around and update references... So, crinkler, cr-linker, compressor-linker, is a linker with an integrated compressor.<br />
<br />
Secondly, crinkler is using a compression method that is far more aggressive and efficient than any old <a href="http://en.wikipedia.org/wiki/Dictionary_coder">dictionary-coder-LZ methods</a> : it's called <b>context modeling coupled with an arithmetic coder</b>. As mentioned in the crinkler's manual, the best place I found to learn about this was <a href="http://mattmahoney.net/dc/">Matt Mahoney resource website</a>. This is definitely the place to start when you want to play with context modeling, as there are lots of sourcecode, previous version of <a href="http://en.wikipedia.org/wiki/PAQ">PAQ </a>program, from which you can learn gradually how to build such a compressor (more particularly in earlier version of the program, when the design was still simple to handle). Building a context-modelling based compressor/decompressor is almost accessible from any developer, but <b>one of the strength of crinkler is its decompressor size </b>: around <b>210-220 bytes</b>, which makes it probably the most efficient and smaller context-modelling decompressor in the world. We will see also that crinkler made one of the simplest choice for a context-modelling compressor, using a semi-static model in order to achieve better compression for 4k of datas, resulting in a less complex decompressor code as well.<br />
<br />
Lastly, <b>crinkler is optimizing the usage of the exe-PE file </b>(which is the Windows Portable Executable format, the binary format of the a windows executable file, official description is available <a href="http://www.microsoft.com/whdc/system/platform/firmware/pecoff.mspx">here</a>). Mostly by removing the standard import table and dll loading in favor of a custom loader that exploit internal windows structure as well as storing function hashing in the header of the PE files to recover dll functions.<br />
<br />
<h2>Compression method</h2><br />
<h3>Arithmetic coding</h3><br />
The whole compression problem in crinkler can be summarized like this: what is the probability of the next bit to compress/decompress to be 1? The better is the probability (meaning by matching the expecting result bit), the better is the compression ratio. Hence, Crinkler needs to be a little bit psychic?!<br />
<br />
First of all, you probably wonder why probability is important here. This is mainly due to one compression technique called <a href="http://en.wikipedia.org/wiki/Arithmetic_coding">arithmetic coding</a>. I won't go into the detail here and encourage the reader to read about the wikipedia article and related links. The main principle of arithmetic coding is its ability to encode into a single number a set of symbols for which you know their probability to occur. The higher the probability is for a known symbol, the lower the number of bits will be required to encode its compressed counterpart. <br />
<br />
At the bit level, things are getting even simpler, since the symbols are only 1 or 0. So if you can provide a probability for the next bit (even if this probability is completely wrong), you are able to encode it through an arithmetic coder.<br />
<br />
A simple binary arithmetic coder interface could look like this:<br />
<pre class="brush: cpp" name="code">/// Simple ArithmeticCoder interface
class ArithmeticCoder {
/// Decode a bit for a given probability.
/// Decode returns the decoded bit 1 or 0
int Decode(Bitstream inputStream, double probabilityForNextBit);
/// Encode a bit (nextBit) with a given probability
void Encode(Bitstream outputStream, int nextBit, double probabilityForNextBit);
}
</pre><br />
And a simple usage of this ArithmeticCoder could look like this:<br />
<pre class="brush: cpp" name="code">// Initialize variables
Bitstream inputCompressedStream = ...;
Bitstream outputStream = ...;
ArithmeticCoder coder;
Context context = ...;
// Simple decoder implem using an arithmetic coder
for(int i = 0; i < numberOfBitsToDecode; i++) {
// Made usage of our psychic alias Context class
double nextProbability = context.ComputeProbability();
// Decode the next bit from the compressed stream, based on this
// probability
int nextBit = coder.Decode( inputCompressedStream, nextProbability);
// Update the psychic and tell him, how much wrong or right he was!
context.UpdateModel( nextBit, nextProbability);
// Output the decoded bit
outputStream.Write(nextBit);
}
</pre><br />
So a Binary Arithmetic Coder is able to compress a stream of bits, if you are able to tell him what's the probability for the next bit in the stream. Its usage is fairly simple, although their implementations are often really tricky and sometimes quite obscure (a real arithmetic implementation should face lots of small problems : renormalization, underflow, overflow...etc.). <br />
<br />
Working at the bit level here wouldn't have been possible 20 years ago, as It requires a tremendous amount of CPU (and memory for the psychic-context) in order to calculate/encode a single bit, but with nowadays computer power, It's less a problem... Lots of implem are working at the byte level for better performance, some of them can work at the bit level while still batching the decoding/encoding results at the byte level. Crinkler doesn't care about this and is working at the bit level, making the arithmetic decoder in less than 20 x86 ASM instructions.<br />
<br />
The C++ pseudo-code for an arithmetic decoder is like this:<br />
<br />
<pre class="brush: cpp" name="code">int ArithmeticCoder::Decode(Bitstream inputStream, double nextProbability) {
int output = 0; // the decoded symbol
// renormalization
while (range < 0x80000000) {
range <<= 1;
value <<= 1;
value += inputStream.GetNextBit();
}
unsigned int subRange = (range * nextProbability);
range = range - subRange;
if (value >= range) { // we have the symbol 1
value = value - range;
range = subRange;
output++; // output = 1
}
return output;
}
</pre><br />
This is almost exactly what is used in crinkler, but this done in only 18 asm instructions! The crinkler arithmetic coder is using a 33 bit precision. The decoder only needs to handle up to 0x80000000 limit renormalization while the encoder needs to work on 64 bit to handle the 33 bit precision. This is much more convenient to work at this precision for the decoder, as it is able to easily detect renormalization (0x80000000 is in fact a negative number. The loop could have been formulated like while (range >= 0), and this is how it is done in asm).<br />
<br />
So the arithmetic coder is the basic component used in crinkler. You will find plenty of arithmetic coder examples on Internet. Even if you don't fully understand the theory behind them, you can use them quite easily. I found for example an interesting project called <a href="http://flavor.sourceforge.net/">flavor</a>, which provides a tool to produce some arithmetic coders code based on a formal description (For example, a <a href="http://flavor.sourceforge.net/samples/bac/tois.htm">32bit precision arithmetic coder </a>description in flavor), pretty handy to understand how things are translated from different coder behaviors.<br />
<br />
But, ok, the real brain here is not the arithmetic coder... but the psychic-context (the Context class above) which is responsible to provide a probability and to update its model based on the previous expectation. This is where a compressor is making the difference.<br />
<br />
<h3>Context modeling - Context mixing</h3><br />
This is one great point about using an arithmetic coder: they can be decoupled from the component responsible to provide the probability for the next symbol. This component is called a <b>context-modeling</b>. <br />
<br />
What is the context? It is whatever data can help your context-modeler to evaluate the probability for the next symbol to occur. Thus, the most obvious data for a compressor-decompressor is to use previous decoded data to update its internal probability table.<br />
<br />
Suppose you have the following sequence of 8 bytes <code>0x7FFFFFFF,0xFFFFFFFF</code> that is already decoded. What will be the next bit? It is certainly to be a 1, and you could bet on it as high as 98% of probability. <br />
<br />
So this is not a surprise that using history of data is the key point for the context modeler to predict next bit (and well, we have to admit that our computer-psychic is not as good as he claims, as he needs to know the past to predict the future!).<br />
<br />
Now that we know that to produce a probability for the next bit, we need to use historic data, how crinkler is using them? Crinkler is in fact maintaining a table of probability, up to 8 bytes + the current bits already read before the next bit. In the context-modeling jargon, it's often called the order (before context modeling, there was technique developped like <a href="http://en.wikipedia.org/wiki/Prediction_by_partial_matching">PPM </a>for Partial Predition Matching and <a href="http://en.wikipedia.org/wiki/Dynamic_Markov_compression">DMC</a> for dynamic markov compression). But crinkler is using not only the last x bytes (up to 8), but sparse mode (as it is mentioned in PAQ compressors), a combination of the last 8 bytes + the current bits already read. Crinkler calls this a model: It is stored into a single byte :<br />
<ul><li>The 0x00 model says that It doesn't use any previous bytes other than the current bits being read.</li>
<li>The 0x80 model says that it is using the previous byte + the current bits being read.</li>
<li>The 0x81 model says that is is using the previous byte and the -8th byte + the current bits being read.</li>
<li>The 0xFF model says that all 8 previous bytes are used</li>
</ul>You probably don't see yet how this is used. We are going to take a simple case here: Use the previous byte to predict the next bit (called the model 0x80). <br />
<br />
Suppose the sequence of datas :<br />
<pre><span style="color: #38761d;">
0xFF, 0x80, 0xFF, 0x85, 0xFF, 0x88, 0xFF</span>, <span style="color: red;">???nextBit???</span>
(0) (1) (2) (3) | => decoder position
</pre><br />
<ul><li>At position 0, we know that 0xFF is followed by bit 1 (0x80 <=> 10000000b). So n0 = 0, n1 = 1 (n0 denotes the number of 0 that follows 0xFF, n1 denotes the number of 1 that usually follows 0xFF)</li>
<li>At position 1, we know that 0xFF is still followed by bit 1: n0 = 0, n1 = 2</li>
<li>At position 2, n0 = 0, n1 = 3</li>
<li>At position 3, we have n0 = 0, n1 = 3, making the probability for one p(1) = (n1 + eps) / ( n0+eps + n1+eps). eps for epsilon, lets take 0.01. We have p(1) = (2+0.01)/(0+0.01 + 2+0.01) = 99,50%</li>
</ul><br />
So we have the probability of 99,50% at position (3) that the next bit is a 1.<br />
<br />
The principle here is simple: For each model and an historic value, we associate n0 and n1, the number of bits found for bit 0 (n0) and bit 1 (n1). Updating those n0/n1 counters needs to be done carefully : a naive approach would be to increment according values when a particular training bit is found... but there is more chance that recent values are more relevant than olders.... Matt Mahoney explained this in <a href="http://mattmahoney.net/dc/paq1.pdf"><i>The PAQ1 Data Compression Program</i></a>, 2002. (Describes PAQ1), and describes how to efficiently update those counters for a non-stationary source of data :<br />
<ul><li>If the training bit is y (0 or 1) then increment ny (n0 or n1).</li>
<li>If n(1-y) > 2, then set n(1-y) = n(1-y) / 2 + 1 (rounding down if odd). </li>
</ul><br />
Suppose for example that n0 = 3 and n1 = 4 and we have a new bit 1. Then n0 will be = n0/2 + 1 = 3/2+1=2 and n1 = n1 + 1 = 5<br />
<br />
Now, we know how to produce a single probability for a single model... but working with a single model (for exemple, only the previous byte) wouldn't be enough to evaluate correctly the next bit. Instead, we need a way to combine different models (different selection of historic data). This is called <b>context-mixing</b>, and this is the real power of context modeling: whatever is your method to collect and calculate a probability, you can, at some point, mix severals estimator to calculate a single probability.<br />
<br />
There are several ways to mix those probabilities. In the pure context-modeling jargon, the model is the way you mix probabilities and for each model, you have a weight :<br />
<ul><li><b>static</b>: you determine the weights whatever the data are.</li>
<li><b>semi-static</b>: you perform a 1st pass over the data to compress to determine the weights for each model, and them a 2nd pass with the best weights</li>
<li> <b>adaptive</b>: weights are updated dynamically as new bits are discovered.</li>
</ul><br />
<b>Crinkler is using a semi-static context-mixing</b> but is somewhat also "semi-adaptive", because It is using different weights for the code of your exe, and the data of your exe, as they have a different binary layout.<br />
<br />
So how this is mixed-up? Crinkler needs to determine the best context-models (the combination of historic data) that It will use, assign for each of those context a weight. The weight is then used to calculate the final probability.<br />
<br />
<br />
For each selected historic model (i) with an associated model weight wi, and ni0/ni1 bit counters, the final probability p(1) is calculated like this :<br />
<br />
<div style="text-align: center;"><pre>p(1) = Sum( wi * ni1 / (ni0 + ni1)) / Sum ( wi )</pre></div><br />
This is exactly what is done in the code above for <code>context.ComputeProbability();</code>, and this is exactly what crinkler is doing.<br />
<br />
In the end, crinkler is selecting a list of models for each type of section in your exe: a set of models for the code section, a set of models for the data section.<br />
<br />
How many models crinkler is selecting? It depends on your data. For example, for <a href="http://code4k.blogspot.com/2010/08/making-of-ergon-4k-pc-intro.html">ergon </a>intro,crinklers is selecting the following models:<br />
<br />
<pre>For the code section:
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Model {0x00,0x20,0x60,0x40,0x80,0x90,0x58,0x4a,0xc0,0xa8,0xa2,0xc5,0x9e,0xed,}
Weight { 0, 0, 0, 1, 2, 2, 2, 2, 3, 3, 3, 4, 6, 6,}
For the data section:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Model {0x40,0x60,0x44,0x22,0x08,0x84,0x07,0x00,0xa0,0x80,0x98,0x54,0xc0,0xe0,0x91,0xba,0xf0,0xad,0xc3,0xcd,}
Weight { 0, 0, 0, 0, 0, 0, 0, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5,}
</pre>(note that in crinkler, the final weight used to multiply n1/n0+n1 is by 2^w, and not wi itself).<br />
<br />
Wow, does it means that crinkler needs to store those datas in your exe. (14 bytes + 20 bytes) * 2 = 68 bytes? Well, crinkler authors are smarter than this! In fact the models are stored, but weights are only store in a single int (32 bits for each section). Yep, a single int to stored those weights? Indeed: if you look at those weights, they are increasing, sometimes they are equal... So they found a clever way to store a compact representation of those weights in a 32 bit form. Starting with a weight of 1, the 32bit weight is shifted by one bit to the left : If this is 0, than the currentWeight doesn't change, if bit is 1, than currentWeight is incremented by 1 : (in this pseudo-code, shift is done to the right)<br />
<br />
<pre class="brush: cpp" name="code">int currentWeight = 1;
int compactWeight = ....;
foreach (model in models) {
if ( compactWeight & 1 )
currentWeigh++;
compactWeight = compactWeight >> 1;
// ... used currentWeight for current model
}
</pre><br />
This way, crinkler is able to store a compact form of pairs (model/weight) for each type of data in your executable (code or pure data).<br />
<br />
<h3>Model selection</h3><br />
Model selection is one of the key process of crinkler. For a particular set of datas, what is the best selection of models? You start with 256 models (all the combinations of the 8 previous bytes) and you need to determine the best selection of models. You have to take into account that each time you are using a model, you need to use 1 byte in your final executable to store this model. Model selection is part of crinkler compressor but is not part of crinkler decompressor. The decompressor just need to know the list of the final models used to compress the data, but doesn't care about intermediate results. On the other hand, the compressor needs to test every combination of model, and find an appropriate weight for each model.<br />
<br />
I have tested several methods in my test code and try to recover the method used in crinkler, without achieving comparable compression ratio... I tried some brute force algo without any success... The selection algorithm is probably a bit clever than the one I have tested, and would probably require to layout mathematics/statistics formulas/combination to select an accurate method.<br />
<br />
Finally, blueberry has given their method (thanks!) <br />
<br />
"<i>To answer your question about the model selection process, it is actually not very clever. We step through the models in bit-mirrored numerical order (i.e. 00, 80, 40, C0, 20 etc.) and for each step do the following:<br />
<br />
- Check if compression improves by adding the model to the current set of models (taking into account the one extra byte to store the model).<br />
<br />
- If so, add the model, and then step through every model in the current set and remove it if compression improves by doing so.<br />
<br />
The difference between FAST and SLOW compression is that SLOW optimizes the model weights for every comparison between model sets, whereas FAST uses a heuristic for the model weights (number of bits set in the model mask). </i>"<br />
<br />
<br />
On the other hand, I tried a fully adaptive context modelling approach, using dynamic weight calculation explained by Matt Mahoney with neural networks and stretch/squash functions (look at PAQ on wikipedia). It was really promising, as I was able to achieve sometimes better compression ratio than crinkler... but at the cost of a decompressor 100 bytes heavier... and even I was able to save 30 to 60 bytes for the compressed data, I was still off by 40-70 bytes... so under 4k, this approach was definitely not as efficient as a semi-static approach chosen by crinkler.<br />
<br />
<h3>Storing probabilities</h3><br />
If you have correctly followed the previous model selection, crinkler is now working with a set of models (selection of history data), for each bit that is found, each model probabilities must be updated...<br />
<br />
But think about it: for example, if to predict the following bit, we are using the probabilities for the 8 previous bytes, it means that for every combination of 8 bytes already found in the decoded data, we would have a pair of n0/n1 counters?<br />
<br />
That would mean that we could have the folowing probabilities to update for the context 0xFF (8 previous bytes):<br />
- "00 00 00 00 c0 00 00 <span style="color: #b45f06;">50 </span><span style="color: #38761d;">00</span>" => some n0/n1<br />
- "00 00 70 00 00 00 00 <span style="color: #b45f06;">F2 </span><span style="color: #38761d;">01</span>" => another n0/n1<br />
- "00 00 00 40 00 00 00 <span style="color: #b45f06;">30 </span><span style="color: #38761d;">02</span>" => another n0/n1 <br />
...etc.<br />
<br />
and if we have other models like 0x80 (previous byte), or 0xC0 (the last 2 previous bytes), we would have also different counters for them:<br />
<table><tbody>
<tr> <td><br />
<div style="text-align: left;">// For model 0x80</div><div style="text-align: left;">- "<span style="color: #38761d;">00</span>" => some n0/n1</div><div style="text-align: left;">- "<span style="color: #38761d;">01</span>" => another n0/n1</div><div style="text-align: left;">- "<span style="color: #38761d;">02</span>" => yet another n0/n1 </div><div style="text-align: left;">...</div></td> <td><br />
<div style="text-align: left;">// For model 0xC0</div><div style="text-align: left;">- "<span style="color: #b45f06;">50 </span><span style="color: #38761d;">00</span>" => some bis n0/n1</div><div style="text-align: left;">- "<span style="color: #b45f06;">F2 </span><span style="color: #38761d;">01</span>" => another bis n0/n1</div><div style="text-align: left;">- "<span style="color: #b45f06;">30 </span><span style="color: #38761d;">02</span>" => yet another bis n0/n1 </div><div style="text-align: left;">...</div></td> </tr>
</tbody></table><br />
From the previous model context, I have slightly over simplified the fact that not only the previous bytes is used, but also the current bits being read. In fact, when we are using for example the model 0x80 (using the previous byte), the context of the historic data is composed not only by the previous byte, but also by the bits being read on the current octet. This implies obviously that for every bit read, there is a different context. Suppose we have the sequence 0x75, 0x86 (in binary 10000110b), the position of the encoded bits is just after the 0x75 value and that we are using the previous byte + the bits currently read:<br />
<br />
First, we start on a byte boundary<br />
- 0x75 with 0 bit (we start with 0) is followed by bit 1 (the 8 of 0x85). The context is 0x75 + 0 bit read<br />
- We read one more bit, we have a new context : 0x75 + bit 1. This context is followed by a 0<br />
- We read one more bit, we have a new context : 0x75 + bit 10. This context is followed by a 0.<br />
...<br />
- We read one more bit, we have a new context : 0x75 + bit 1000011, that is followed by a 0 (and we are ending on a byte boundary).<br />
<br />
Reading 0x75 followed by 0x86, with a model using only the previous byte, we finally have 8 context with their own n0/n1 to store in the probability table.<br />
<br />
As you can see, It is obvious that It's difficult to store all context found (.i.e for each single bit decoded, there is a different context of historic bytes) and their respective exact probability counters, without exploding the RAM. Moreover if you think about the number of models that are used by crinkler: 14 types of different historic previous bytes selection for ergon's code!<br />
<br />
This kind of problem is often handled using a hashtable while handling collisions. This is what is done in some of the PAQ compressors. <b>Crinkler is also using an hashtable to store counter probabilities</b>, with the association context_history_of_bytes = > (n0/n1), but It is not handling collision in order to keep minimal the size of the decompressor. As usual, the hash function used by crinkler is really tiny while still giving really good results.<br />
<br />
So instead of having the association between context_history_of_bytes => n0/n1, we are using a hashing function, hash(context_history_of_bytes) => n0/n1. Then, the dictionary that is storing all those associations needs to be correctly dimensioned, large enough, to store as much as possible associations found while decoding/encoding the data.<br />
<br />
Like in PAQ compressors, <b>crinkler is using one byte for each counter</b>, meaning that n0 and n1 together are taking 16 bit, 2 bytes. So if you instruct crinkler to use a hashtable of 100Mo, It will be possible to store 50 millions of different keys, meaning different historic context of bytes and their respective probability counters. There is a little remark about crinkler and the byte counter: in PAQ compressors, limits are handled, meaning that if a counter is going above 255, It will stuck to 255... but crinkler made the choice to not test the limits in order to keep the code smaller (although, that would take less than 6 bytes to test the limit). What is the impact of this choice? Well, if you know crinkler, you are aware that crinkler doesn't handle large section of "zeros" or whatever empty initialized data. This is just because the probabilities are looping from 255 to 0, meaning that you jump from a 100% probability (probably accurate) to almost a 0% probability (probably wrong) every 256 bytes. Is this really hurting the compression? Well, It would hurt a lot if crinkler was used for larger executable, but in a 4k, It's not hurting so much (although, It could hurt if you really have large portions of initialized data). Also, not all the context are reseted at the same time (a 8 byte context will not probably reset as often as a 1 byte context), so it means that final probability calculation is still accurate... while there is a probability that is reseted, other models with their own probabilities are still counting there... so this is not a huge issue.<br />
<br />
What happens also if the hash for a different context is giving the same value? Well, the model is then updating the wrong probability counters. If the hashtable is too small the probability counters may really be too much disturbed and they would provide a less accurate final probability. But if the hashtable is large enough, collisions are less likely to happen.<br />
<br />
Thus, it is quite common to use a hashtable as large as 256 to 512Mo if you want, although 256Mo is often enough, but the larger is your hashtable, the less are collisions, the more accurate is your probability. Recall from the beginning of this post, and you should understand now why "crinkler can take several hundreds of megabytes to decompress"... simply because of this hashtable that store all the probabilities for the next bit for all models combination used.<br />
<br />
If you are familiar with crinkler, you already know the option to find a best possible hashsize for an initial hashtable size and a number of tries (hashtries option). This part is responsible to test different size of hashtable (like starting from 100Mo, and reducing the size by 2 bytes 30 times, and test the final compression) and test final compression result. This is a way to empirically reduce collision effects by selecting the hashsize that is giving the better compression ratio (meaning less collisions in the hash). Although this option is only able to help you save a couple of bytes, no more.<br />
<br />
<br />
<h3>Data reordering and type of data</h3><br />
Reordering or organizing differently the data to have a better compression is one of the common technique in compression methods. Sometimes for example, Its better to store deltas of values than to store values themselves...etc.<br />
<br />
<b>Crinkler is using this principle to perform data reordering</b>. At the linker level, crinkler has access to portion of datas and code, and is able to move those portions around in order to achieve a better compression ratio. This is really easy to understand : suppose that you have a series initialized zero values in your data section. If those values are interleaved with non zero values, the counter probabilities will switch from "there are plenty of zero there" to "ooops, there are some other datas"... and the final probability will balance between 90% to 20%. Grouping data that are similar is a way to improve the overall probability correctness.<br />
<br />
This part is the most time consuming, as It needs to move and arrange all portions of your executable around, and test which arrangement is giving the best compression result. But It's paying to use this option, as you may be able to save 100 bytes in the end just with this option.<br />
<br />
One thing that is also related to data reordering is the way <b>crinkler is handling separately the binary code and the data of your executable</b>. Why?, because their binary representation is different, leading to a completely different set of probabilities. If you look at the selected models for ergon, you will find that code and data models are quite different. Crinkler is using this to achieve better performance here. In fact, crinkler is compressing completely separately the code and the datas. Code has its own models and weights, Data another set of models and weights. What does it means internally? Crinkler is using a set of model and weights to decode the code section of your exectuable. Once finished, It will erase the probability counters stored in the hashtable-dictionary, and go to the data section, with new models and weights. Reseting all counters to 0 in the middle of decompressing is improving compression by a factor of 2-4%, which is quite impressive and valuable for a 4k (around 100 to 150 bytes).<br />
<br />
I found that even with an adaptive model (with a neural networks dynamically updating the weights), It is still worth to reset the probabilities between code and data decompression. In fact, reseting the probabilities is an empirical way to instruct the context modeling that datas are so different that It's better to start from scratch with new probability counters. If you think about it, an improved demo compressor (for larger exectuable, for example under 64k) could be clever to detect those portions of datas that are enough different that It would be better to reset the dictionary than to keep it as it is.<br />
<br />
There is just one last thing about weights handling in crinkler. When decoding/encoding, It seems that crinkler is artificially increasing the weights for the first discovered bit. This little trick is improving compression ratio by about 1 to 2% which is not bad. Having higher weights at the beginning enable to have a better response of the compressor/decompressor, even If it doesn't still have enough data to compute a correct probability. Increasing the weights is helping the compression ratio at cold start.<br />
<br />
Crinkler is also able to transform the x86 code for the executable part to improve compression ratio. This technique is widely used and consist of replacing relative jump (conditionnal, function calls...etc.) to absolute jump, leading to a better compression ratio.<br />
<br />
<h2>Custom DLL LoadLibrary and PE file optimization</h2><br />
In order to strip down the size of an executable, It's necessary to exploit as much as possible the organization of a PE file.<br />
<br />
First thing that crinkler is using is that lots of part in a PE files are not used at all. If you want to know how a windows executable PE files can be reduced, I suggest you read <a href="http://www.phreedom.org/solar/code/tinype/">Tiny PE</a> article, which is a good way to understand what is actually used by a PE loader. Unlike the Tiny PE sample, where the author is moving the PE header to the dos header, crinkler made the choice to use this unused place to store hash values that are used to reference DLL functions used.<br />
<br />
This trick is called import by hashing and is quite common in intro's compressor. Probably what make crinkler a little bit more advanced is that to perform the "GetProcAddress" (which is responsible to get the pointer to a function from a function name), <b>crinkler is navigating inside internal windows process structure in order to directly get the address of the functions from the in-memory import table. </b>Indeed, you won't find any import section table in a crinklerized executable. Everything is re-discovered through internal windows structures. Those structures are not officially documented but you can find some valuable information around, most notably <a href="http://www.alex-ionescu.com/part1.pdf">here</a>.<br />
<br />
If you look at crinkler's code stored in the crinkler import section, which is the code injected just before the intros start, in order to load all dll functions, you will find those cryptics calls like this:<br />
<pre>//
(0) MOV EAX, FS:[BX+0x30]
(1) MOV EAX, [EAX+0xC]
(2) MOV EAX, [EAX+0xC]
(3) MOV EAX, [EAX]
(4) MOV EAX, [EAX]
(5) MOV EBP, [EAX+0x18]
</pre><br />
<br />
This is done by going through internal structures: <br />
<ul><li><b>(0)</b> first crinklers gets a pointer to the "PROCESS ENVIRONMENT BLOCK (PEB)" with the instruction <code>MOV EAX, FS:[BX+0x30]</code>. EAX is now pointing to the PEB </li>
</ul><pre><span style="color: blue;">Public Type</span> <span style="background-color: #990000;"><span style="background-color: white; color: #990000;">PEB</span></span> </pre><pre>InheritedAddressSpace As <span style="color: #990000;">Byte</span>
ReadImageFileExecOptions As <span style="color: #990000;">Byte</span>
BeingDebugged As <span style="color: #990000;">Byte</span>
Spare As <span style="color: #990000;">Byte</span>
Mutant As <span style="color: #990000;">Long</span>
SectionBaseAddress As <span style="color: #990000;">Long</span>
ProcessModuleInfo As <span style="color: #990000;">Long </span>‘ <b><span style="color: #38761d;">// <---- PEB_LDR_DATA</span></b>
ProcessParameters As <span style="color: #990000;">Long </span>‘ <span style="color: #38761d;">// RTL_USER_PROCESS_PARAMETERS</span>
SubSystemData As <span style="color: #990000;">Long</span>
ProcessHeap As <span style="color: #990000;">Long</span>
... struct continue</pre><br />
<ul><li><b>(1)</b> Then it gets a pointer to the "ProcessModuleInfo/PEB_LDR_DATA" <code>MOV EAX, [EAX+0xC]</code></li>
</ul><pre><span style="color: #0b5394;"><span style="color: blue;">Public Type</span> </span><span style="color: #990000;">_PEB_LDR_DATA</span>
Length As <span style="color: #990000;">Integer</span>
Initialized As <span style="color: #990000;">Long</span>
SsHandle As <span style="color: #990000;">Long</span>
InLoadOrderModuleList As <span style="color: #990000;">LIST_ENTRY</span> <b><span style="color: #38761d;">// <---- LIST_ENTRY InLoadOrderModuleList</span></b>
InMemoryOrderModuleList As <span style="color: #990000;">LIST_ENTRY</span>
InInitOrderModuleList As <span style="color: #990000;">LIST_ENTRY</span>
EntryInProgress As <span style="color: #990000;">Long</span>
<span style="color: blue;">End Type</span></pre><br />
<ul><li><b>(2)</b> Then it gets a pointer to get a pointer to the next "InLoadOrderModuleList/LIST_ENTRY" <code>MOV EAX, [EAX+0xC]</code>.</li>
</ul><pre><span style="color: blue;">Public Type</span> <span style="color: #990000;">LIST_ENTRY</span> Flink As <span style="color: #990000;">LIST_ENTRY</span>
Blink As <span style="color: #990000;">LIST_ENTRY</span>
<span style="color: blue;">End Type</span></pre><br />
<ul><li><b>(3) and (4)</b> Then it navigates through the LIST_ENTRY linked list <code>MOV EAX, [EAX]</code>. This is done 2 times. First time, we get a pointer to the NTDLL.dll, second with get a pointer to the KERNEL.DLL. Each LIST_ENTRY is in fact followed by the structure LDR_MODULE : </li>
</ul><br />
<pre><span style="color: blue;">Public Type</span> LDR_MODULE
InLoadOrderModuleList As LIST_ENTRY
InMemoryOrderModuleList As LIST_ENTRY
InInitOrderModuleList As LIST_ENTRY
BaseAddress As Long
EntryPoint As Long
SizeOfImage As Long
FullDllName As UNICODE_STRING
BaseDllName As UNICODE_STRING
Flags As Long
LoadCount As Integer
TlsIndex As Integer
HashTableEntry As LIST_ENTRY
TimeDateStamp As Long
LoadedImports As Long
EntryActivationContext As Long ‘ // ACTIVATION_CONTEXT
PatchInformation As Long
<span style="color: blue;">End Type</span></pre><br />
Then from the BaseAddress of the Kernel.dll module, crinkler is going to the section where functions are already loaded in memory. From there, the first hashed function that is stored by crinkler is LoadLibrary function. After this, crinkler is able to load all the depend dll and navigate through the import tables, recomputing the hash for all functions names for dependent dlls, and is trying to match the hash stored in the PE header. If a match is found, then the function entry point is stored.<br />
<br />
This way, crinkler is able to call some OS functions stored in the Kernel.DLL, without even linking explicitly to those DLL, as they are automatically loaded whenever a DLL is loaded. Thus achieving a way to import all functions used by an intro with a custom import loader.<br />
<br />
<h2 id="results">Compression results </h2><br />
So finally, you may ask, how much crinkler is good at compressing? How does it compare to other compression method? How does look like the entropy in a crinklerized exe?<br />
<br />
I'll take the example of Ergon exe. You can already find a <a href="http://code4k.blogspot.com/2010/08/making-of-ergon-4k-pc-intro.html#statistics">detailed analysis </a>for this particular exe.<br />
<br />
<h3>Comparison with other compression methods</h3><br />
In order to make a fair comparison between crinkler and other compressors, I have used the data that are actually compressed by crinkler after the reordering of code and data (This is done by unpacking a crinklerized ergon.exe and extracting only the compressed data). This comparison is accurate in that all compressors are using exactly the same data.<br />
<br />
In order also to be fair with crinkler, the size of 3652 is not taking into account the PE header + the crinkler decompressor code (which in total is 432 bytes for crinkler).<br />
<br />
To perform this comparison, I have only used 7z which has at least 3 interesting methods to test against :<br />
<ul><li>Standard Deflate Zip</li>
<li>PPMd with 256Mo of dictionary</li>
<li>LZMA with 256Mo of dictionary</li>
</ul>I have also included a comparison with a more advanced packing method from Matt Mahoney resource, <a href="http://mattmahoney.net/dc/paq.html#paq8">Paq8l </a>which is one of the version of PAQ methods, using neural networks and several context modeling methods.<br />
<br />
<div align="center"><table border="0" cellpadding="0" cellspacing="0" style="width: 470px;"><col style="width: 68pt;" width="91"></col> <col style="width: 131pt;" width="175"></col> <col style="width: 48pt;" width="64"></col> <col style="width: 105pt;" width="140"></col> <tbody>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="background: none repeat scroll 0% 0% rgb(192, 80, 77); border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; height: 15pt; text-decoration: none; width: 68pt;" width="91">Program</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(192, 80, 77); border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 131pt;" width="175">Compression Method</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(192, 80, 77); border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 48pt;" width="64">Size in bytes</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(192, 80, 77); border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 105pt;" width="140">Ratio vs Crinkler</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">none</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">uncompressed</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9796</td> <td class="xl65" style="border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;"><br />
</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">crinkler</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">ctx-model 256Mo</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3652</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(124, 197, 124); border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">+0,00%</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">7z</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">deflate 32Ko</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4526</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(248, 105, 107); border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">+23,93%</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">7z</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">PPMd 256Mo</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4334</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">+18,67%</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color rgb(192, 80, 77); border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">7z</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">LZMA 256Mo</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid none none; border-width: 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4380</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(254, 204, 127); border-color: rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">+19,93%</td> </tr>
<tr height="20" style="height: 15pt;"> <td class="xl65" height="20" style="border-color: rgb(192, 80, 77) -moz-use-text-color rgb(192, 80, 77) rgb(192, 80, 77); border-style: solid none solid solid; border-width: 0.5pt medium 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">Paq8l</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color; border-style: solid none; border-width: 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">dyn-ctx-model 256Mo</td> <td class="xl65" style="border-color: rgb(192, 80, 77) -moz-use-text-color; border-style: solid none; border-width: 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3521</td> <td class="xl67" style="background: none repeat scroll 0% 0% rgb(99, 190, 123); border-color: rgb(192, 80, 77) rgb(192, 80, 77) rgb(192, 80, 77) -moz-use-text-color; border-style: solid solid solid none; border-width: 0.5pt 0.5pt 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">-3,59%</td> </tr>
</tbody></table></div><br />
As you can see, <b>crinkler is far more efficient than any of the "standard" compression method </b>(Zip, PPMd, LZMA). I'm not even talking about the fact that <b>a true comparison would be to include the decompressor size, so the ratio should certainly be worse for all standard methods!</b><br />
<br />
Paq8l is of course slightly better... but if you take into account that Paq8l decompressor is itself an exe of 37Ko... compare to the 220 byte of crinkler... you should understand now how much crinkler is highly efficient in its own domain! (remember? 4k!)<br />
<br />
<h3>Entropy</h3><br />
In order to measure the entropy of crinkler, I have developed a very small program in C# that is displaying the entropy of an exe. From green color (low entropy, less bits necessary to encode this information) to red color (high entropy, more bits necessary to encode this information).<br />
<br />
I have done this on 3 different ergon executable :<br />
<ul><li>The uncompressed ergon.exe (28Ko). It is the standard output of a binary exe with MSVC++ 2008.</li>
<li>The raw-crinklerized ergon.exe extracted code and data section, but not compressed (9796 bytes)</li>
<li>The final crinklerized ergon.exe file (4070 bytes)</li>
</ul><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiut2MUBTNQVPk1BsP3zCyAzTmWflYaT-a6Ro4DOlyoZwnxmmExzBTUvL9VxLBQXp7UUwecNYhqyt9vVKxJa7Jkhyw8hRdRBRzP8fxtNANkmVLQ8D9-OFOH8EzEaIp2Sb2vvh9ZQ9cVMr8/s1600/entropy_ergon_plain_exe.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiut2MUBTNQVPk1BsP3zCyAzTmWflYaT-a6Ro4DOlyoZwnxmmExzBTUvL9VxLBQXp7UUwecNYhqyt9vVKxJa7Jkhyw8hRdRBRzP8fxtNANkmVLQ8D9-OFOH8EzEaIp2Sb2vvh9ZQ9cVMr8/s320/entropy_ergon_plain_exe.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ergon standard exe entropy</td></tr>
</tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJfNahX-7YKBYlBKGYnqDa-wT6j49F0_E59KvNdWOhpwCaHCf6YNFgbjssAm8IDz94V0uARxYUVYfzFcp9NFzGIVlKZMYZSjKfdS0jSfjauD-CDmTOKwtfg0M4gT1ryPoo6lnam2BC8-s/s1600/entropy_ergon_codedata_reordered_and_pack.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJfNahX-7YKBYlBKGYnqDa-wT6j49F0_E59KvNdWOhpwCaHCf6YNFgbjssAm8IDz94V0uARxYUVYfzFcp9NFzGIVlKZMYZSjKfdS0jSfjauD-CDmTOKwtfg0M4gT1ryPoo6lnam2BC8-s/s320/entropy_ergon_codedata_reordered_and_pack.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ergon code and data crinklerized, uncompressed reordered data </td></tr>
</tbody></table><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE6mARVsZGhRPLblGhEZ2xyUwleJYYRiER6xe44tQeDAUl90luUI4K6tRAHa85oEVHVWS5g6z0yFL3bNQQwTWreCEQxp7jqwEH-JZF5e1MFw_w-PJDW5V4amIIi498r0LSOh5lrKWCUGw/s1600/entropy_ergon_crinklerized.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE6mARVsZGhRPLblGhEZ2xyUwleJYYRiER6xe44tQeDAUl90luUI4K6tRAHa85oEVHVWS5g6z0yFL3bNQQwTWreCEQxp7jqwEH-JZF5e1MFw_w-PJDW5V4amIIi498r0LSOh5lrKWCUGw/s320/entropy_ergon_crinklerized.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Ergon executable crinklerized</td></tr>
</tbody></table>As expected, <b>the entropy is fairly massive in a crinklerized exe</b>. Compare with the waste of information in a standard windows executable. Also, you can appreciate how much is important the reordering and packing of data (no compression) that is perform by crinkler.<br />
<br />
<h3>Some notes about the x86 crinkler decompressor asm code</h3><br />
I have often talked about how much crinkler decompressor is truly a piece of x86 art. It is hard to describe the technique used here, there are lots of x86 standard optimization and some really nice trick. Most notably:<br />
<ol><li><b>using all the registers</b></li>
<li><b>using </b><b>intensively the stack to save/restore all the registers with pushad/popad x86</b>. This is for example done (1 + number_of_model) per bit. If you have 15 models, there will be a total of 16 pushad/popad instructions for a single bit to be decoded! You may wonder why making so many pushes? Its the only way to efficiently use all the registers (rule #1) without having to store particular registers in a buffer. Of course, push/pop instruction is also used at several places in the code as well.</li>
<li>As a result of 1) and 2), apart from the hash dictionnary,<b> no intermediate structure are used to perform the context modeling calculation</b>.</li>
<li><b>Deferred conditional jump</b>: Usually, when you perform some conditional testing with x86, this is often immediately followed by a conditional jump (like cmp eax, 0; jne go_for_bla). In crinkler, sometimes, a conditionnal test is done, and is used several instruction laters. (for example. cmp eax,0; push eax; mov eax, 5; jne go_for_bla <---- this is using the result of cmp eax,0 comparison). It makes the code to read a LOT harder. Sometimes, the conditional is even used after a direct jump! This is probably one part of crinkler's decompressor that impressed me the most. This is of course something quite common if you are programming heavily optimized-size x86 asm code... you need to know of course which instructions is not modifying CPU flags in order to achieve this kind of optimization!</li>
</ol><br />
<h2>Final words</h2><br />
I would like to apologize for the lack of charts, pictures to explain a little bit how things are working. This article is probably still obscure for a casual reader, and should be considered as a draft version. This was a quick and dirty post. I wanted to write this for a long time, so here it is, not perfect as it should be, but this may be improved in future versions! <br />
<br />
As you can see, crinkler is really worth to look at. The effort to make it so efficient is impressive and there is almost no doubt that there won't be any other crinkler competitor for a long time! At least for a 4k executable. Above 4k, I'm quite confident that there are still lots of area that could be improved, and probably kkrunchy is far from being the ultimate packer under 64k... Still, if you want a packer, you need to code it, and that's not so trivial!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com19tag:blogger.com,1999:blog-1076643699683521890.post-52900584516728360962010-12-01T00:17:00.006+11:002011-03-15T23:43:20.782+11:00Official release of SharpDX 1.0<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYwkwXUGxwT3TscF1XiqGvOJw4Q4a0pcr5iraSENv8M7bxYeB7I86HJPWnVsd_4nvRAEwoOebQLzDgZB6rsusLTGUHlEI1siCeaiDxrT1Anbc-poWr98WI97JHeNg7DunrD-JWzpdmGfg/s1600/sharpdx_logo.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYwkwXUGxwT3TscF1XiqGvOJw4Q4a0pcr5iraSENv8M7bxYeB7I86HJPWnVsd_4nvRAEwoOebQLzDgZB6rsusLTGUHlEI1siCeaiDxrT1Anbc-poWr98WI97JHeNg7DunrD-JWzpdmGfg/s200/sharpdx_logo.png" width="200" /></a></div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiU3Y6-5MqHZxDnRp81fNVrS3mwrHVSsZme_EjxkGRTVTqRC2VqDGTP9WOsJzeQFUdUT8CNhhu1pRS1pwGjBkgtenh-rDr8v0UV4580yEsd9PN5iDSFaOO2bJStaKJqE-zCYP6o3wxTUKg/s1600/sharpdx_logo_large.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><br />
</a><br />
After three months of intense development, I'm really excited to announce the availability of <a href="http://code.google.com/p/sharpdx/">SharpDX 1.0</a> , a new platform independent .Net managed DirectX API, directly generated from DirectX SDK headers.<br />
<br />
This first version can be considered as stable. The Direct3D10 / Direct3D10.1 API has been entirely tested on a large 3D engine that was using previously SlimDX (thanks patapom!). <a href="http://code.google.com/p/sharpdx/wiki/MigrateFrom">Migration </a>was quite straightforward, with tiny minor changes to the engine's code.<br />
<br />
<br />
The key features and benefits of this new API are: <br />
<ul><li><b>API is generated from DirectX SDK headers</b> : meaning <b>a complete and reliable API</b> and <b>an easy support for future API</b>. </li>
<li><b>Full support for the following DirectX API</b>: </li>
<ul><li>Direct3D10 </li>
<li>Direct3D10.1 </li>
<li>Direct3D11 </li>
<li>Direct2D1 (including custom rendering, tessellation callbacks) </li>
<li>DirectWrite (including custom client callbacks) </li>
<li>D3DCompiler </li>
<li>DXGI </li>
<li>DXGI 1.1 </li>
<li>DirectSound </li>
<li>XAudio2 </li>
<li>XAPO </li>
<li>An integrated math API directly ported from SlimMath </li>
</ul><li><b>Pure managed .NET API, platform independent</b> : assemblies are compiled with AnyCpu target. You can run your code on a x64 or a x86 machine with the same assemblies, without recompiling your project. </li>
<li><b>Lightweight <a href="http://code.google.com/p/sharpdx/wiki/AssemblyDlls">individual assemblies</a></b> : a core assembly - SharpDX - containing common classes and an assembly for each subgroup API (Direct3D10, Direct3D11, DXGI, D3DCompiler...etc.). Assemblies are also lightweight. </li>
<li><b>C++/CLI Speed</b> : the framework is using a genuine way to avoid any C++/CLI while still achieving comparable performance. </li>
<li><b>API naming convention mostly compatible with SlimDX API</b>. </li>
<li><b>Raw DirectX object life management</b> : No overhead of ObjectTable or RCW mechanism, the API is using direct native management with classic COM method "Release". </li>
<li><b>Easily mergeable / obfuscatable</b> : If you need to obfuscate SharpDX assemblies, they are easily obfusctable due to the fact the framework is not using any mixed assemblies. You can also merge SharpDX assemblies into a single exe using with tool like <a href="http://research.microsoft.com/en-us/people/mbarnett/ilmerge.aspx" rel="nofollow">ILMerge</a>. </li>
</ul>You will also find a growing collection of samples in the <a href="http://code.google.com/p/sharpdx/wiki/SamplesGallery">Samples Gallery</a> of SharpDX. Most notably with some additional support for Direct2D1 and DirectWrite client callbacks.<br />
<br />
Instead of providing a monolithic assembly, SharpDX is providing lightweight individual and interdependent assemblies. All SharpDX assemblies are dependent from the core SharpDX assembly. You just need to add the required assemblies to your project, without embedding the whole DirectX API stack. Here is a chart that explains SharpDX assembly dependencies:<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://sharpdx.googlecode.com/svn/trunk/Doc/SharpDXAssemblyDependencies.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="457" src="http://sharpdx.googlecode.com/svn/trunk/Doc/SharpDXAssemblyDependencies.png" width="640" /></a></div><br />
Next versions will provide support for DirectInput, XInput, X3DAudio, XACT3.<br />
<br />
<h3>About performance</h3>Someone asked me how SharpDX compares to SlimDX in terms of performance. Here is a micro-benchmark on two methods, ID3D10Device1::GetFeatureLevel (alias Device.FeatureLevel) and ID3D10Device::CheckCounterInfo (alias Device.GetCounterCapabilities). <br />
<br />
The test consist of 100,000,000 calls on each methods (inside a for, with (10 calls to device.FeatureLevel) * 10,000,000 times) and is repeated 10 times and averaged. Repeated two times.<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="width: 540px;"><colgroup><col style="width: 159pt;" width="212"></col> <col style="width: 77pt;" width="103"></col> <col style="width: 63pt;" width="84"></col> <col style="width: 106pt;" width="141"></col> </colgroup><tbody>
<tr height="20" style="height: 15pt;"> <td height="20" style="background: none repeat scroll 0% 0% rgb(75, 172, 198); border-color: -moz-use-text-color white white -moz-use-text-color; border-style: none solid solid none; border-width: medium 0.5pt 1.5pt medium; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; height: 15pt; text-align: center; text-decoration: none; width: 159pt;" width="212">Method</td> <td style="background: none repeat scroll 0% 0% rgb(75, 172, 198); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-align: center; text-decoration: none; width: 77pt;" width="103">SlimDX</td> <td style="background: none repeat scroll 0% 0% rgb(75, 172, 198); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-align: center; text-decoration: none; width: 63pt;" width="84">SharpDX</td> <td style="background: none repeat scroll 0% 0% rgb(75, 172, 198); border-color: -moz-use-text-color -moz-use-text-color white white; border-style: none none solid solid; border-width: medium medium 1.5pt 0.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-align: center; text-decoration: none; width: 106pt;" width="141">SharpDX vs SlimDX</td> </tr>
<tr height="20" style="height: 15pt;"> <td height="20" style="background: none repeat scroll 0% 0% rgb(183, 222, 232); border-color: white white white -moz-use-text-color; border-style: solid solid solid none; border-width: 0.5pt 0.5pt 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">device.FeatureLevel</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(183, 222, 232); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">3700</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(183, 222, 232); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">3650</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(78, 222, 54); border-color: white -moz-use-text-color white white; border-style: solid none solid solid; border-width: 0.5pt medium 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">1,37%</td> </tr>
<tr height="20" style="height: 15pt;"> <td height="20" style="background: none repeat scroll 0% 0% rgb(218, 238, 243); border-color: white white -moz-use-text-color -moz-use-text-color; border-style: solid solid none none; border-width: 0.5pt 0.5pt medium medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">device.GetCounterCapabilities()</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(218, 238, 243); border-color: white white -moz-use-text-color; border-style: solid solid none; border-width: 0.5pt 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">4684</td> <td class="xl65" style="background: none repeat scroll 0% 0% rgb(218, 238, 243); border-color: white white -moz-use-text-color; border-style: solid solid none; border-width: 0.5pt 0.5pt medium; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">4259</td> <td class="xl66" style="background: none repeat scroll 0% 0% rgb(78, 222, 54); border-color: white -moz-use-text-color -moz-use-text-color white; border-style: solid none none solid; border-width: 0.5pt medium medium 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-align: center; text-decoration: none;">9,98%</td> </tr>
</tbody></table><br />
For FeatureLevel, the test was sometimes around +/-0.5%.<br />
For GetCounterCapabilities(), the main difference between SharpDX and SlimDX implementation is that SlimDX perform a copy from the native struct to .Net struct while SharpDX is directly passing a pointer to the .Net struct.<br />
<br />
This test is of course a micro benchmark and doesn't reflect a real-world usage. Some part of the API could be in favor of SlimDX, but I'm pretty confident that SharpDX is much more consistent in the way structures are passed to the native functions, avoiding as much as possible marshaling structures that doesn't need any custom marshaling (unlike SlimDX that is performing most of a time a marshaling between .Net/Native structure, besides they are binary compatible).<br />
<br />
<h3>Next?</h3>Finally, I'm going to be able to use this project to make some demos with it! Next target is to develop a XNA like based framework based on SharpDX.Direct3D11.<br />
<br />
Stay tuned!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com13tag:blogger.com,1999:blog-1076643699683521890.post-52125803950929442162010-11-18T21:06:00.007+11:002010-12-07T23:53:29.122+11:00SharpDX, a new managed .Net DirectX API availableIf you have followed my previous work on <a href="http://code4k.blogspot.com/2010/10/managed-netc-direct3d-11-api-generated.html">a new .NET API for Direct3D 11</a>, I proposed SlimDX team this solution for the v2 of their framework, joined their team around one month ago, and I was actively working to widen the coverage of the DirectX API. I have been able to extend the API coverage almost up to the whole API, being able to develop Direct2D samples, as well as XAudio2 and XAPO samples using it. But due to some incompatible directions that the SlimDX team wanted to follow, I have decided to release also my work under a separate project called <a href="http://code.google.com/p/sharpdx/">SharpDX</a>. Now, you may wonder why I'm releasing this new API under a separate project from SlimDX?<br />
<br />
Well, I have been working really hard on this from the beginning of September, and I explained why in my previous post about Direct3D 11. I have checked-in lots of code under the v2 branch on SlimDX, while having lots of discussion with the team (mostly Josh which is mostly responsible for v2) on their devel mailing list. The reason I'm leaving SlimDX team is that It was in fact not clear for me that I was not enrolled as part of the decision for the v2 directions, although I was bringing a whole solution (by "whole", I mean a large proof of concept, not something robust, finished). At some point, Josh told me that Promit, Mike and himself, co-founders of SlimDX, were the technical leaders of this project and they would have the last word on the direction as well as for decisions on the v2 API.<br />
<br />
Unfortunately, I was not expecting to work in such terms with them, considering that I had already made 100% of the whole engineering prototype for the next API. From the last few days, we had lots of -small- technical discussions, but for some of them, I clearly didn't agree about the decisions that were taken, whatever the arguments I was trying to give to them. This is a bit of disappointment for me, but well, that's life of open source projects. This is their project and they have other plans for it. So, I have decided to release the project on my own with <a href="http://code.google.com/p/sharpdx/">SharpDX </a>although you will see that the code is also currently exactly the same on the v2 branch of SlimDX (of course, because until yesterday, I was working on the SlimDX v2 branch).<br />
<br />
But things are going to change for both projects : SlimDX is taking the robust way (for which I agree) but with some decisions that I don't agree (in terms of implementation and direction). Although, as It may sound weird, SharpDX is not intended to compete with SlimDX v2 : They have clearly a different scope (supporting for example Direct3D 9, which I don't really care in fact), different target and also different view on exposing the API and a large existing community already on SlimDX. So SharpDX is primarily intended for my own work on demomaking. Nothing more. I'm releasing it, because SlimDX v2 is not going to be available soon, even for an alpha version. On my side, I'm considering that the current state (although far to be as clean as It should be) of the SharpDX API is usable and I'm going to use it on my own, while improving the generator and parser, to make the code safer and more robust.<br />
<br />
So, I did lots of work to bring new API into this system, including :<br />
<ul><li>Direct3D 10</li>
<li>Direct3D 10.1</li>
<li>Direct3D 11</li>
<li>Direct2D 1</li>
<li>DirectWrite</li>
<li>DXGI</li>
<li>DXGI 1.1</li>
<li>D3DCompiler</li>
<li>DirectSound</li>
<li>XAudio2</li>
<li>XAPO</li>
</ul>And I have been working also on some nice samples, for example using Direct2D and Direct3D 10, including the usage of the tessellate Direct2D API, in order to see how well It works compared to the gluTessellation methods that are most commonly used. You will find that the code is extremely simple in SharpDX to do such a thing :<br />
<pre class="brush: csharp" name="code">using System;
using System.Drawing;
using SharpDX.Direct2D1;
using SharpDX.Samples;
namespace TessellateApp
{
/// <summary>
/// Direct2D1 Tessellate Demo.
/// </summary>
public class Program : Direct2D1DemoApp, TessellationSink
{
EllipseGeometry Ellipse { get; set; }
PathGeometry TesselatedGeometry{ get; set; }
GeometrySink GeometrySink { get; set; }
protected override void Initialize(DemoConfiguration demoConfiguration)
{
base.Initialize(demoConfiguration);
// Create an ellipse
Ellipse = new EllipseGeometry(Factory2D,
new Ellipse(new PointF(demoConfiguration.Width/2, demoConfiguration.Height/2), demoConfiguration.Width/2 - 100,
demoConfiguration.Height/2 - 100));
// Populate a PathGeometry from Ellipse tessellation
TesselatedGeometry = new PathGeometry(Factory2D);
GeometrySink = TesselatedGeometry.Open();
// Force RoundLineJoin otherwise the tesselated looks buggy at line joins
GeometrySink.SetSegmentFlags(PathSegment.ForceRoundLineJoin);
// Tesselate the ellipse to our TessellationSink
Ellipse.Tessellate(1, this);
// Close the GeometrySink
GeometrySink.Close();
}
protected override void Draw(DemoTime time)
{
base.Draw(time);
// Draw the TextLayout
RenderTarget2D.DrawGeometry(TesselatedGeometry, SceneColorBrush, 1, null);
}
void TessellationSink.AddTriangles(Triangle[] triangles)
{
// Add Tessellated triangles to the opened GeometrySink
foreach (var triangle in triangles)
{
GeometrySink.BeginFigure(triangle.Point1, FigureBegin.Filled);
GeometrySink.AddLine(triangle.Point2);
GeometrySink.AddLine(triangle.Point3);
GeometrySink.EndFigure(FigureEnd.Closed);
}
}
void TessellationSink.Close()
{
}
[STAThread]
static void Main(string[] args)
{
Program program = new Program();
program.Run(new DemoConfiguration("SharpDX Direct2D1 Tessellate Demo"));
}
}
}
</pre><br />
This simple example is producing the following ouput :<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiC_Q0PRuf20K-blLz6YciB8ezy22lN8Uss4zx4h5O6TM_voaTEUpnJvbQvvOigUgbuLc_Fg3KucjATuepzBmzPCIkWdYjRgsR8arsEWpk4OKnzQlqfHC9RVXlSQKZsxXGiAU9-rE59Z-Y/s1600/SharpDXDirect2DTessellate.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiC_Q0PRuf20K-blLz6YciB8ezy22lN8Uss4zx4h5O6TM_voaTEUpnJvbQvvOigUgbuLc_Fg3KucjATuepzBmzPCIkWdYjRgsR8arsEWpk4OKnzQlqfHC9RVXlSQKZsxXGiAU9-rE59Z-Y/s320/SharpDXDirect2DTessellate.png" width="320" /></a></div><br />
which is pretty cool, considering the amount of code (although the Direct3D 10 and D2D initialization part would give a larger code), I found this to be much simpler than the gluTessellation API.<br />
<br />
You will find also some other samples, like the XAudio2 ones, generating a synthesized sound with the usage of the reverb, and even some custom XAPO sound processors!<br />
<br />
You can <a href="http://code.google.com/p/sharpdx/">grab those samples on SharpDX </a>code repository (there is a SharpDXBinAndSamples.zip with a working solutions with all the samples I have been developing so far, with also MiniTris sample from SlimDX).xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com3tag:blogger.com,1999:blog-1076643699683521890.post-87968284850828413632010-11-03T10:46:00.009+11:002012-08-11T12:01:32.106+11:00Hacking Direct2D to use directly Direct3D 11 instead of Direct3D 10.1 API<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJZyR1jT8gP4OHshImMfLdUNXP5o4XfEILZMaCDZnVAs4ECGJ-T3_2ekbxejz6Ne7JUpoeluMgkdxC9pa6vVMtNoZfNiDF5df3IQXA4mH8-KYax0PUXLrJ1-cXuxfxJxHztSQztrBw2GU/s1600/D2D1ToD3D11.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJZyR1jT8gP4OHshImMfLdUNXP5o4XfEILZMaCDZnVAs4ECGJ-T3_2ekbxejz6Ne7JUpoeluMgkdxC9pa6vVMtNoZfNiDF5df3IQXA4mH8-KYax0PUXLrJ1-cXuxfxJxHztSQztrBw2GU/s1600/D2D1ToD3D11.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJZyR1jT8gP4OHshImMfLdUNXP5o4XfEILZMaCDZnVAs4ECGJ-T3_2ekbxejz6Ne7JUpoeluMgkdxC9pa6vVMtNoZfNiDF5df3IQXA4mH8-KYax0PUXLrJ1-cXuxfxJxHztSQztrBw2GU/s320/D2D1ToD3D11.png" width="320" /></a></div>
<strong>Disclaimer about this hack</strong>: This hack was nothing more than a proof of concept and I *really* don't have time to dig into any kind of bugs related to it.<br/>
<br/>
<span style="font-size: x-small;">
[Edit]13 Jan 2011, After Windows Update <a href="http://support.microsoft.com/kb/2454826">KB2454826</a>, this hack was not working. I have patched the sample to make it work again. Of course, you shouldn't consider this hack for anykind of production use. Use the standard DXGI shared sync keyed mutex instead. This hack is just for fun![/Edit]
</span><br />
<br />
<br />
If you know Direct3D 11 and Direct 2D - they were released almost at the same time - you already know that <b>there is a huge drawback to use Direct 2D : It's in fact only working with Direct3D 10.1 API</b> (although It's working with older hardware thanks to the new feature level capability of the API).<br />
<br />
From a coding user point of view, this is really disappointing that such a good API doesn't rely on the latest Direct3D API... moreover when you know that the Direct3D 11 API is really close to the Direct3D 10.1 API... In the end, more work are required for a developer that would like to work with Direct3D 11, as It doesn't have any more Text API for example, meaning that in D3D11, you have to do it yourself, which isn't a huge task itself, if you go to the easy precalculated-texture-of-fonts generated by some GDI+ calls or whatever, but still... this is annoying specially when you need to display some information/FPS on the screen and you can't wait to build a nice font-texture-based system...<br />
<br />
I'm not completely fair with Direct2D interoperability with Direct3D 11 : there is in fact a well known solution proposed by one guy from DirectX Team that imply the use of <a href="http://www.gamedev.net/community/forums/topic.asp?topic_id=547920">DXGI mutex to synchronized a surface shared between D3D10.1 and D3D11</a>. I was expecting this issue to be solved in some DirectX SDK release this year, but It seems that there is <a href="http://blogs.msdn.com/b/directx/archive/2009/12/18/direct2d-and-directwrite.aspx?PageIndex=2#comments">no plan to release in the near future an update for Direct2D</a> (see my question in the comments and the anwser...)... WP7 and XNA are probably getting much more attention here...<br />
<br />
So last week, I took some time on the Direct2D API and found that <b>It's in fact fairly easy to hack Direct2D and redirect all the D3D10.1 API calls to a real Direct3D 11 instance</b>... and this is a pretty cool news! Here is the story of this little hack...<br />
<br />
<a name='more'></a><br />
<h3>
How Direct2D is accessing your already instantiated D3D10.1 device?</h3>
<br />
In order to use Direct2D with a renderable D3D10 texture2D, you need to query the IDXGISurface from your ID3D10Texture2D object, something like this:<br />
<pre class="brush: cpp" name="code">IDXGISurface* surface;
// Create a Texture2D (or use SwapChain backbuffer)
d3d10Device->CreateTexture2D(&texture2DDesc, 0, &texture2D);
// Query the DXGI Surface associated with the D3D10.1 Texture2D
texture2D->QueryInterface(__uuidof(IDXGISurface), &surface);
// Create a D2D Render target from the D3D10 Texture2D through the associated DXGISurface
d2dFactory->CreateDxgiSurfaceRenderTarget(
surface,
&props,
&d2dRenderTarget
);
</pre>
So starting from this CreateDxgiSurfaceRenderTarget call, Direct2D is somehow able to get back your D3D10.1 instance and is able to use it to submit drawcalls / create textures... etc. In order to find how Direct2D is getting an instance of ID3D10Device1, I have first implemented a Proxy IDXGISurface that was responsible to embed the real DXGI Surface and delegate all the calls for it...while being able to track down how Direct2D is getting back this ID3D10Device1 : <br />
<br />
<ul>
<li>After the surface enters the CreateDxgiSurfaceRenderTarget, Direct2D is querying the IDXGIDevice through the GetDevice method on the IDXGISurface</li>
<li>From the IDXGIDevice, Direct2D is calling QueryInterface with the IID of the ID3D10Device interface (surprisingly not the ID3D10Device1)</li>
</ul>
And bingo! Being able to give your own implementation of the ID3D10Device to Direct2D... and you are able to redirect all the D3D10 calls to a Direct3D 11 device/context with a simple proxy implementing ID3D10Device1 methods!<br />
<br />
<h3>
Interoperability between D3D10.1 and D3D11 API</h3>
<br />
Migrating from D3D10/D3D10.1 to D3D11 API is quite straightforward and even have <a href="http://msdn.microsoft.com/en-us/library/ff476190%28v=VS.85%29.aspx">a dedicated paper on msdn</a>. For the purpose of this quick hack, I didn't implement proxies for the whole D3D10 API... but I have instead focused my work on how is used the D3D10 API from D2D and what are the real methods/structures used that are not binary compatible between D3D10 and D3D11.<br />
<br />
In the end, I have developped 5 proxies :<br />
<ul>
<li>a Proxy for IDXGISurface interface, in order to hack the GetDevice method and return my own proxy for IDXGIDevice</li>
<li>a Proxy for IDXGIDevice interface in order to hack the QueryInterface method and return my own proxy for ID3D10Device1</li>
<li>a Proxy for the ID3D10Device1 interface</li>
<li>a Proxy for the ID3D10Texture2D interface</li>
<li>a Proxy for the ID3D10Buffer interface</li>
</ul>
For the ID3D10Device1 interface, most of the methods are redirecting the calls directly to the device (ID3D11Device) or context (ID3D11DeviceContext). I didn't bother to implement proxies for most of the parameters, because even if they are not always binary compatible, returned objects are only used as reference and are not called directly. Suppose for example the proxy implementation for VSGetShader (which is used by Direct2D for saving the D3D10 pipeline state) :<br />
<pre class="brush: cpp" name="code">virtual void STDMETHODCALLTYPE VSGetShader(
/* [annotation] */
__out ID3D10VertexShader **ppVertexShader) {
context->VSGetShader((ID3D11VertexShader**)ppVertexShader, 0, 0);
}
</pre>
<br />
A Real proxy would have to wrap the ID3D11VertexShader inside a ID3D10VertexShader proxy... but because Direct2D (and this is not a surprise) is only using VSGetShader to later call VSSetShader (in order to restore the saved states, or to set it's own vertex/pixel shaders), It doesn't call any method on the ID3D10VertexShader instance... meaning that we can give it back directly a ID3D11VertexShader without performing any - costly - conversion.<br />
<br />
For instance, most of the ID3D10Device1 proxy methods are like the previous one, a simple redirection to a D3D11 Device or DeviceContext... easy!<br />
<br />
I was only forced to implement custom proxies for some incompatible structures... or returned object instance that are effectively used by Direct2D (like ID3D10Buffer and ID3D10Texture2D).<br />
<br />
For example, the ID3D10Device::CreateBuffer proxy methods is implemented like this :<br />
<br />
<pre class="brush: cpp" name="code">virtual HRESULT STDMETHODCALLTYPE CreateBuffer(
/* [annotation] */
__in const D3D10_BUFFER_DESC *pDesc,
/* [annotation] */
__in_opt const D3D10_SUBRESOURCE_DATA *pInitialData,
/* [annotation] */
__out_opt ID3D10Buffer **ppBuffer) {
D3D11_BUFFER_DESC desc11;
*((D3D10_BUFFER_DESC*)&desc11) = *pDesc;
// StructureByteStride field is new in D3D11
desc11.StructureByteStride = 0;
// Returns our ID3D10Buffer proxy instead of the real one
ProxyID3D10Buffer* buffer = new ProxyID3D10Buffer();
buffer->device = this;
*ppBuffer = buffer;
HRESULT result = device()->CreateBuffer(&desc11, (D3D11_SUBRESOURCE_DATA*)pInitialData, (ID3D11Buffer**)&buffer->backend);
CHECK_RETURN(result);
// return S_OK;
}
</pre>
<br />
There was also just a few problems with 2 incompatible structures between D3D10_VIEWPORT/D3D11_VIEWPORT (D3D11 is using floats instead of ints!) and D3D10_BLEND_DESC/D3D11_BLEND_DESC... but the proxy methods were easy to implement:<br />
<br />
<pre class="brush: cpp" name="code">virtual void STDMETHODCALLTYPE RSSetViewports(
/* [annotation] */
__in_range(0, D3D10_VIEWPORT_AND_SCISSORRECT_OBJECT_COUNT_PER_PIPELINE) UINT NumViewports,
/* [annotation] */
__in_ecount_opt(NumViewports) const D3D10_VIEWPORT *pViewports) {
// Perform conversion between D3D10_VIEWPORT and D3D11_VIEWPORT
D3D11_VIEWPORT viewports[16];
for(int i = 0; i < NumViewports; i++) {
viewports[i].TopLeftX = pViewports[i].TopLeftX;
viewports[i].TopLeftY = pViewports[i].TopLeftY;
viewports[i].Width = pViewports[i].Width;
viewports[i].Height = pViewports[i].Height;
viewports[i].MinDepth = pViewports[i].MinDepth;
viewports[i].MaxDepth = pViewports[i].MaxDepth;
}
context->RSSetViewports(NumViewports, (D3D11_VIEWPORT*)viewports);
}
</pre>
<br />
Even if I haven't performed any performance timing measurement, the cost of those proxy methods should be almost unnoticeable... and probably much more lightweight than using mutex synchronization between D3D10 and D3D11 devices!<br />
<br />
<h3>
Plug-in the proxies</h3>
<br />
In the end, I have managed to put those proxies in a single .h/.cpp with an easy API to plug the proxy. The sequence call before passing the DXGISurface to Direct2D should then be like this:<br />
<br />
<pre class="brush: cpp" name="code">d3d11Device->CreateTexture2D(&offlineTextureDesc, 0, &texture2D);
// Create a Proxy DXGISurface from Texture2D compatible with Direct2D
IDXGISurface* surface = Code4kCreateD3D10CompatibleSurface(d3d11Device, d3d11DeviceContext, texture2D);
d2dFactory->CreateDxgiSurfaceRenderTarget(
surface,
&props,
&d2dRenderTarget
);
</pre>
<br />
And that's all! You will find attached a project with the sources. Feel free to test it and let me know if you are encountering any issues with it. Also, the code is far from being 100% safe/robust... It's a quick hack. For example, I have not checked carefully that my proxies behaves well with AddRef/Release... but that should be fine.<br />
<br />
So far, It's seems to work well on the whole Direct2D API... I have even been able to use DirectWrite with Direct2D... using Direct3D 11, without any problem. There is only one issue : PIX won't be able to debug Direct2D over Direct3D 11... because It seems that Direct2D is performing some additional method calls (D3D10CreateStateBlocks) that are incompatible with the lightweight proxies I have developed... In order to be fully supported, It would be necessary to implement all the proxies for all the interfaces returned by ID3D10Device1... But this is a sooo laborious task that by that time, we can expect to have Direct2D fully working with Direct3D 11 provided from DirectX Team itself!<br />
<br />
Also from this little experience, I can safely confirm that It shouldn't take more than one day for one guy from the Direct2D team to patch existing Direct2D code in order to use Direct3D 11... as it is <i>much</i> easier to do this on the original code than going to the proxy road as I did! ;)<br />
<br />
<hr />
<br />
You can grab the VC++ 2010 project from here : <a href="http://xoofx.com/upload/D2D1ToD3D11.7z">D2D1ToD3D11.7z</a><br />
<br />
This sample is only saving a "test.png" image using Direct2D API over Direct3D11.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com18tag:blogger.com,1999:blog-1076643699683521890.post-54059696723218170892010-10-26T02:28:00.004+11:002010-12-02T20:50:46.390+11:00Implementing an unmanaged C++ interface callback in C#/.NetEver wanted to implement a C++ interface callback in a managed C# application? Well, although that's not so hard, this is a solution that you will probably hardly find over the Internet... the most common answer you will get is that it's not possible to do it or you should use C++/CLI in order to achieve it... In fact, in C#, you can only implement a C function delegate through the use of <a href="http://msdn.microsoft.com/en-US/library/system.runtime.interopservices.marshal.getfunctionpointerfordelegate.aspx"><code>Marshal.GetFunctionPointerForDelegate</code></a> but you won't find anything like <code>Marshal.GetInterfacePointerFromInterface</code>. You may wonder why do I need such a thing? <br />
<br />
In my previous post about implementing <a href="http://code4k.blogspot.com/2010/10/managed-netc-direct3d-11-api-generated.html">a new DirectX fully managed API</a>, I forgot to mention the case of interfaces callbacks. There are not so many cases in Direct3D 11 API where you need to implement a callback. You will more likely find more use-cases in audio APIs like XAudio2, but in Direct3D 11, afaik, you will only find 3 interfaces that are used for callback:<br />
<ul><li><a href="http://msdn.microsoft.com/en-us/library/ff728746%28VS.85%29.aspx">ID3DInclude</a> which is used by D3DCompiler API in order to provide a callback for includes while using preprocessor or compiler API (see for example <a href="http://msdn.microsoft.com/en-us/library/dd607324%28VS.85%29.aspx">D3DCompile</a>).</li>
<li><a href="http://msdn.microsoft.com/en-us/library/ff476644%28v=VS.85%29.aspx">ID3DX11DataLoader </a>and <a href="http://msdn.microsoft.com/en-us/library/ff476648%28v=VS.85%29.aspx">ID3DX11DataProcessor</a>, which are used by some D3DX functions in order to perform asynchronous loading/processing of texture resources. The nice thing about C# is that those interfaces are useless, as it is <i>much </i>easier and trivial to directly implement them in C# instead</li>
</ul>So I'm going to take the example of ID3DInclude, and how It has been successfully implemented for the <a href="http://code.google.com/p/sharpdx">SharpDX</a>.<br />
<a name='more'></a><br />
<h3>Memory layout of a C++ object implementing pure virtual methods</h3><br />
If you know how a C++ interface with pure methods is layout in memory, that's fairly easy to imagine how to hack C# to provide such a thing, but if you don't, here is a quick summary:<br />
<br />
For example, the ID3DInclude C++ interface is declared like this :<br />
<pre class="brush: cpp" name="code">// Interface declaration
DECLARE_INTERFACE(ID3DInclude)
{
STDMETHOD(Open)(THIS_ D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) PURE;
STDMETHOD(Close)(THIS_ LPCVOID pData) PURE;
};
</pre><br />
DECLARE_INTERFACE is a Windows macro that is defined in ObjBase.h and will expand the previous declaration in C++ like this:<br />
<br />
<pre class="brush: cpp" name="code">struct ID3DInclude {
virtual HRESULT __stdcall Open(D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) = 0;
virtual HRESULT __stdcall Close(LPCVOID pData) = 0;
}; </pre><br />
Implementing and using this interface in C++ is straightforward:<br />
<pre class="brush: cpp" name="code">struct MyIncludeCallback : public ID3DInclude {
virtual HRESULT __stdcall Open(D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) {
/// code for Open callback
}
virtual HRESULT __stdcall Close(LPCVOID pData) {
/// code for Close callback
}
};
// Usage
ID3DInclude* include = new MyIncludeCallback();
// Compile a shader and use our Include provider
D3DCompile(..., include, ...);
</pre><br />
The hack here is to clearly understand how is layout in memory an instance of ID3DInclude through the <a href="http://en.wikipedia.org/wiki/Virtual_method_table">Virtual Method Table (VTBL)</a>... Oh, it's really funny to see that the Wikipedia article doesn't use any visual table to represent a virtual table... ok, let's remedy it. If you look at the memory address of an instanciated object, you will find an indirect pointer:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDfgS0muCwkby8pHrTQG-h5eJyp2dfeeTNaEy1LK2r2phVrOcCg5InSDx895qrqIDlXOtAkbjDvZgLTHtqkjC7_alTrrzM3vovLP4piSz7Ftw7f0ELF8hpmopL40gjtMRpGbRPY8EBSwY/s1600/VTBL.png" style="margin-left: auto; margin-right: auto;" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. Virtual Method Table layout in memory</td></tr>
</tbody></table><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDfgS0muCwkby8pHrTQG-h5eJyp2dfeeTNaEy1LK2r2phVrOcCg5InSDx895qrqIDlXOtAkbjDvZgLTHtqkjC7_alTrrzM3vovLP4piSz7Ftw7f0ELF8hpmopL40gjtMRpGbRPY8EBSwY/s1600/VTBL.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"></a></div>So from the pointer to a C++ object implementing pure virtual methods, you will find that the first value is a pointer to a VTBL which is shared among the same type of object (here <code>MyIncludeCallback</code>).<br />
<br />
Then in the VTBL, the first value is a pointer to the Open() method implementation in memory. The second to the Close() method.<br />
<br />
According to the calling convention, how does look the declaration of this Open() function, if we had to impleement it in pure C?<br />
<pre class="brush: cpp" name="code">HRESULT __stdcall MyOpenCallbackFunction(void* thisObject, D3D_INCLUDE_TYPE IncludeType, LPCSTR pFileName, LPCVOID pParentData, LPCVOID *ppData, UINT *pBytes) {
/// code for Open callback
}
</pre>Simply add a "this object" as the 1st parameter of the callback function (which represents a pointer to the MyIncludeCallback instance in memory) and you have a callback at the function level!<br />
<br />
You should understand now how we can easily hack this to provide a C++ interface callback in C#<br />
<br />
<h3>Translation to the C#/.Net world</h3><br />
The solution is fairly simple. In order to be able to pass a C++ Interface callback implemented in C# to an unmanaged function, we need to replicate how the unmanaged world is going to call the unmanaged functions and how It does expect to have an interface layout in memory.<br />
<br />
First, we need to define the ID3DInclude interface in pure C#:<br />
<pre class="brush: csharp" name="code">public partial interface Include
{
/// <summary>
/// A user-implemented method for opening and reading the contents of a shader #include file.
/// </summary>
/// <param name="type">A <see cref="SlimDX2.D3DCompiler.IncludeType"/>-typed value that indicates the location of the #include file. </param>
/// <param name="fileName">Name of the #include file.</param>
/// <param name="parentStream">Pointer to the container that includes the #include file.</param>
/// <param name="stream">Stream that is associated with fileName to be read. This reference remains valid until <see cref="SlimDX2.D3DCompiler.Include.Close"/> is called.</param>
/// <unmanaged>HRESULT Open([None] D3D_INCLUDE_TYPE IncludeType,[None] const char* pFileName,[None] LPCVOID pParentData,[None] LPCVOID* ppData,[None] UINT* pBytes)</unmanaged>
//SlimDX2.Result Open(SlimDX2.D3DCompiler.IncludeType includeType, string fileNameRef, IntPtr pParentData, IntPtr dataRef, IntPtr bytesRef);
void Open(IncludeType type, string fileName, Stream parentStream, out Stream stream);
/// <summary>
/// A user-implemented method for closing a shader #include file.
/// </summary>
/// <remarks>
/// If <see cref="SlimDX2.D3DCompiler.Include.Open"/> was successful, Close is guaranteed to be called before the API using the <see cref="SlimDX2.D3DCompiler.Include"/> interface returns.
/// </remarks>
/// <param name="stream">This is a reference that was returned by the corresponding <see cref="SlimDX2.D3DCompiler.Include.Open"/> call.</param>
/// <unmanaged>HRESULT Close([None] LPCVOID pData)</unmanaged>
void Close(Stream stream);
}
</pre><br />
Clearly, this is not exactly what we have in C++... but this is how we would use it... through the usage of Stream. An implementation of this interface would provide a Stream for a particular file to include (most of a time, that could be as simple as <code>stream = new FileStream(fileName)</code>).<br />
<br />
This interface is public in the C#/.Net API... but internally we are going to use a wrapper of this interface that is going to create manually the object layout in memory as well as the VTBL. This is done in this simple constructor:<br />
<br />
<pre class="brush: csharp" name="code">/// <summary>
/// Internal Include Callback
/// </summary>
internal class IncludeCallback
{
public IntPtr NativePointer;
private Include _callback;
private OpenCallBack _openCallBack;
private CloseCallBack _closeCallback;
public IncludeCallback(Include callback)
{
_callback = callback;
// Allocate object layout in memory
// - 1 pointer to VTBL table
// - following that the VTBL itself - with 2 function pointers for Open and Close methods
_nativePointer = Marshal.AllocHGlobal(IntPtr.Size * 3);
// Write pointer to vtbl
IntPtr vtblPtr = IntPtr.Add(_NativePointer, IntPtr.Size);
Marshal.WriteIntPtr(_NativePointer, vtblPtr);
_openCallBack = new OpenCallBack(Open);
Marshal.WriteIntPtr(vtblPtr, Marshal.GetFunctionPointerForDelegate(_openCallBack ));
_closeCallBack = new CloseCallBack(Close);
Marshal.WriteIntPtr(IntPtr.Add(vtblPtr, IntPtr.Size), Marshal.GetFunctionPointerForDelegate(_closeCallBack));
}
</pre><br />
You can clearly see from the previous code that we are allocating a an unmanaged memory that will hold the object VTBL pointer and the VTBL itself... Because we don't need to make 2 allocation (one for the object's vtbl_ptr/data, one for the vtbl), we are laying out the VTBL just after the object itself, like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyFXQRZYkDBTry08SizNUiYq4NVIYQwRTuHFDCbmXGu2MnHJVF-oKE6uX1Lrcnp8iEwKAzI1Vh711zIsoosoo1ApqWe_z2dFVgUTMaGOBG4TXrRXfIRIYNQeRLobgz0YVQlZmEl0oCqj4/s1600/VTBL2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyFXQRZYkDBTry08SizNUiYq4NVIYQwRTuHFDCbmXGu2MnHJVF-oKE6uX1Lrcnp8iEwKAzI1Vh711zIsoosoo1ApqWe_z2dFVgUTMaGOBG4TXrRXfIRIYNQeRLobgz0YVQlZmEl0oCqj4/s1600/VTBL2.png" /></a></div><br />
The declaration of the C# delegates are then straightforward from the C++ declaration:<br />
<pre class="brush: csharp" name="code">[UnmanagedFunctionPointer(CallingConvention.StdCall)]
private delegate SlimDX2.Result OpenCallBack(IntPtr thisPtr, SlimDX2.D3DCompiler.IncludeType includeType, IntPtr fileNameRef, IntPtr pParentData, ref IntPtr dataRef, ref int bytesRef);
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
private delegate SlimDX2.Result CloseCallBack(IntPtr thisPtr, IntPtr pData);
</pre>You just have to implement the Open and Close method in the wrapper and redirect the calls to the managed Include callback, <i>et voila</i>!<br />
<br />
Then after, when calling an unmanaged function that required this callback, you just have to wrap an Include instance with the callback like this:<br />
<pre class="brush: csharp" name="code">Include myIncludeInstance = ... new ...;
IncludeCallback callback = new IncludeCallback(callback);
// callback.NativePointer is a pointer to the object/vtbl allocated structure
D3D.Compile(..., callback.NativePointer, ...);
</pre><br />
Of course, the IncludeCallback is not visible from the public API but is used internally. From a public interface POV, here is how you would use it:<br />
<pre class="brush: csharp" name="code">using System;
using System.IO;
using SlimDX2.D3DCompiler;
namespace TestCallback
{
class Program
{
class MyIncludeCallBack : Include
{
public void Open(IncludeType type, string fileName, Stream parentStream, out Stream stream)
{
stream = new FileStream(fileName, FileMode.Open);
}
public void Close(Stream stream)
{
stream.Close();
}
}
static void Main(string[] args)
{
var include = new MyIncludeCallBack();
string value = ShaderBytecode.PreprocessFromFile("test.fx", null, include);
Console.WriteLine(value);
}
}
}
</pre><br />
You can have a look at the complete source code <a href="http://code.google.com/p/sharpdx/source/browse/trunk/Source/SharpDX/ComObjectCallback.cs">here</a>.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com2tag:blogger.com,1999:blog-1076643699683521890.post-82285392928080649502010-10-23T21:47:00.008+11:002012-10-11T12:26:39.984+11:00High performance memcpy gotchas in C#<span style="font-size: x-small;">(Edit 8 Jan 2011: Update protocol test with Buffer.BlockCopy)</span><br />
<span style="font-size: x-small;">(Edit 11 Oct 2012: Please vote for <a href="https://connect.microsoft.com/VisualStudio/feedback/details/766977/il-bytecode-method-cpblk-badly-implemented-by-x86-clr" target="_blank">the x86 cpblk deficiency on Microsoft Connect</a>)</span><br />
Following my last post about an interesting use of the "cpblk" IL instruction as an unmanaged memcpy replacement, I have to admit that I didn't take the time to carefully verify that performance is actually better. Well, I was probably too optimistic... so I have made some tests and the results are very surprising and not expected to be like these...<br />
<br />
<h3>
The memcpy protocol test in C#</h3>
<br />
When dealing with 3D calculations, large buffers of textures, audio synthesizing or whatever requires a memcpy and interaction with unmanaged world, you will most notably end up with a call to an unmanaged functions like this one:<br />
<br />
<pre class="brush: csharp" name="code">[DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false), SuppressUnmanagedCodeSecurity]
public static unsafe extern void* CopyMemory(void* dest, void* src, ulong count);
</pre>
<br />
In this test, I'm going to compare this implementation with 4 challengers :<br />
<ul>
<li>The cpblk IL instruction</li>
<li>A handmade memcpy function</li>
<li>Array.Copy, although It's not relevant because they don't have the same scope. Array.Copy is managed only for arrays only while memcpy is used to copy portion of datas between managed-unmanaged as well as unmanaged-unmanaged memory.</li>
<li>Marshal.Copy, same as Array.Copy</li>
<li>Buffer.BlockCopy, which is working on managed array but is working with a byte size block copy. </li>
</ul>
The test is performing a series of memcpy with different size of block : from 4 bytes to 2Mo. The interesting part is to run this test on a x86 and x64 mode. Both tests are running on the same Windows 7 OS x64, same machine Intel Core I5 750 (2.66Ghz). The CLR used for this is the Runtime v4.0.30319.<br />
<br />
The naive handmade memcpy is nothing more than this code (not to be the best implem ever but at least safe for any kind of buffer size):<br />
<br />
<pre class="brush: csharp" name="code">static unsafe void CustomCopy(void * dest, void* src, int count)
{
int block;
block = count >> 3;
long* pDest = (long*)dest;
long* pSrc = (long*)src;
for (int i = 0; i < block; i++)
{
*pDest = *pSrc; pDest++; pSrc++;
}
dest = pDest;
src = pSrc;
count = count - (block << 3);
if (count > 0)
{
byte* pDestB = (byte*) dest;
byte* pSrcB = (byte*) src;
for (int i = 0; i < count; i++)
{
*pDestB = *pSrcB; pDestB++; pSrcB++;
}
}
}</pre>
<br />
<h3>
Results</h3>
For the x86 architecture, results are expressed as a throughput in Mo/s - higher is better, blocksize is in bytes :<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="width: 631px;"><colgroup><col style="width: 49pt;" width="65"></col> <col style="width: 50pt;" width="66"></col> <col style="width: 65pt;" width="86"></col> <col style="width: 84pt;" width="112"></col> <col span="2" style="width: 77pt;" width="102"></col> <col style="width: 74pt;" width="98"></col> </colgroup><tbody>
<tr height="21" style="height: 15.75pt;"> <td class="xl70" height="21" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: windowtext windowtext white; border-style: solid; border-width: 1pt 1pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; height: 15.75pt; text-decoration: none; width: 49pt;" width="65">BlockSize</td> <td class="xl75" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white windowtext; border-style: none solid solid; border-width: medium 0.5pt 1.5pt 1pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 50pt;" width="66">x86-cpblk</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 65pt;" width="86">x86-memcpy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 84pt;" width="112">x86-CustomCopy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 77pt;" width="102">x86-Array.Copy</td> <td class="xl74" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color windowtext white white; border-style: none solid solid; border-width: medium 1pt 1.5pt 0.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 77pt;" width="102">x86-Marshal.Copy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 74pt;" width="98">x86-BlockCopy</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">4</td> <td align="right" class="xl76" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border-color: windowtext white white windowtext; border-style: solid; border-width: 1pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">146</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 112, 108); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">458</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 112, 108); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">470</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 105, 107); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">85</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 105, 107); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">81</td> <td align="right" class="xl78" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border-color: windowtext windowtext white white; border-style: solid; border-width: 1pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">150</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">8</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(248, 108, 107); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">294</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 119, 109); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">843</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 124, 110); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1122</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">168</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">167</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(248, 109, 107); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">298</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">16</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(248, 114, 108); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">587</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 133, 112); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1628</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 139, 113); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1904</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 109, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">306</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 109, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">327</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(248, 114, 108); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">577</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">32</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(248, 121, 110); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">950</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 138, 113); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1876</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 163, 118); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3184</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 115, 108); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">631</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 113, 108); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">558</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(249, 123, 110); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1079</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">64</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(249, 130, 111); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1451</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 165, 118); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3316</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 183, 122); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4295</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 126, 111); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1205</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 123, 110); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1059</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(249, 140, 113); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1981</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">128</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 145, 114); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2245</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 200, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5161</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 194, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4848</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 144, 114); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2176</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 139, 113); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1933</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(251, 166, 118); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3386</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">256</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(252, 184, 122); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4353</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 203, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5333</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 172, 120); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3699</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 166, 118); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3386</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(253, 203, 125); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5333</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">512</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(197, 219, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13617</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 206, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5517</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 209, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5663</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 228, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6666</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(252, 234, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7441</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">1024</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(197, 219, 129); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13617</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(140, 202, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">20000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 228, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6666</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(211, 223, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">12075</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(236, 230, 131); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9275</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">2048</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(151, 205, 126); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">18823</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(99, 190, 123); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">24615</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7191</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(237, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9142</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(168, 210, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16842</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(233, 229, 131); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9552</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">4096</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 158, 117); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2922</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7529</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 209, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5663</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(225, 227, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">10491</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(220, 225, 130); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11034</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">8192</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 159, 117); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2990</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7804</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 210, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5714</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(218, 225, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11228</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7441</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(215, 224, 130); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11636</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">16384</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 156, 116); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2857</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 233, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7901</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 208, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5614</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(237, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9142</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7619</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(226, 227, 131); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">10322</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">32768</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 147, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2379</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 229, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6736</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 203, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5333</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(246, 233, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8101</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 228, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6666</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">65536</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 147, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2379</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6808</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 205, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5470</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(254, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6808</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">131072</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 150, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2509</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(160, 208, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">17777</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 212, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5818</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(246, 233, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8101</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(160, 208, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">17777</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(246, 233, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8101</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">262144</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 150, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2500</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(215, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11636</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 204, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5423</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7111</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">524288</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 150, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2539</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 204, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5423</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7111</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7111</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">1048576</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 150, 115); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2539</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 205, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5470</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7111</td> </tr>
<tr height="21" style="height: 15.75pt;"> <td align="right" class="xl72" height="21" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext windowtext; border-style: solid; border-width: 0.5pt 1pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15.75pt; text-decoration: none;">2097152</td> <td align="right" class="xl67" style="background: none repeat scroll 0% 0% rgb(250, 150, 115); border-color: white white windowtext windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2529</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(253, 203, 125); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5333</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(220, 225, 130); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11034</td> <td align="right" class="xl69" style="background: none repeat scroll 0% 0% rgb(254, 232, 131); border-color: white windowtext windowtext white; border-style: solid; border-width: 0.5pt 1pt 1pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6881</td> </tr>
</tbody></table>
<br />
<br />
<br />
<br />
For the x64 architecture:<br />
<br />
<br />
<table border="0" cellpadding="0" cellspacing="0" style="width: 654px;"><colgroup><col style="width: 54pt;" width="72"></col> <col style="width: 50pt;" width="66"></col> <col style="width: 65pt;" width="86"></col> <col style="width: 84pt;" width="112"></col> <col style="width: 77pt;" width="102"></col> <col style="width: 89pt;" width="118"></col> <col style="width: 74pt;" width="98"></col> </colgroup><tbody>
<tr height="21" style="height: 15.75pt;"> <td class="xl70" height="21" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: windowtext windowtext white; border-style: solid; border-width: 1pt 1pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; height: 15.75pt; text-decoration: none; width: 54pt;" width="72">BlockSize2</td> <td class="xl75" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white windowtext; border-style: none solid solid; border-width: medium 0.5pt 1.5pt 1pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 50pt;" width="66">x64-cpblk</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 65pt;" width="86">x64-memcpy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 84pt;" width="112">x64-CustomCopy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color white white; border-style: none solid solid; border-width: medium 0.5pt 1.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 77pt;" width="102">x64-Array.Copy</td> <td class="xl74" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color windowtext white white; border-style: none solid solid; border-width: medium 1pt 1.5pt 0.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 89pt;" width="118">x64-Marshal.Copy</td> <td class="xl73" style="background: none repeat scroll 0% 0% rgb(79, 129, 189); border-color: -moz-use-text-color -moz-use-text-color white white; border-style: none none solid solid; border-width: medium medium 1.5pt 0.5pt; color: white; font-family: Calibri; font-size: 11pt; font-weight: 700; text-decoration: none; width: 74pt;" width="98">x64-BlockCopy</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">4</td> <td align="right" class="xl76" style="background: none repeat scroll 0% 0% rgb(248, 114, 108); border-color: windowtext white white windowtext; border-style: solid; border-width: 1pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">583</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 109, 107); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">346</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 114, 108); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">599</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 105, 107); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">99</td> <td align="right" class="xl77" style="background: none repeat scroll 0% 0% rgb(248, 105, 107); border-color: windowtext white white; border-style: solid; border-width: 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">111</td> <td align="right" class="xl78" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border-color: windowtext windowtext white white; border-style: solid; border-width: 1pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">219</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">8</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(249, 131, 112); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1509</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 117, 109); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">770</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 138, 113); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1876</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 107, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">212</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 106, 107); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">224</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(248, 111, 108); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">469</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">16</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(250, 153, 116); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2689</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 130, 111); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1451</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(251, 165, 118); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3316</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 111, 108); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">417</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 110, 108); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">422</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(248, 118, 109); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">903</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">32</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(252, 191, 123); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4705</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 153, 116); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">2666</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 196, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 118, 109); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">802</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 117, 109); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">864</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(249, 132, 112); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1739</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">64</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 193, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4812</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7272</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 132, 112); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1568</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 132, 112); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">1748</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(250, 159, 117); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3350</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">128</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(200, 219, 129); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13333</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(246, 233, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8101</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(238, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9014</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 159, 117); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3004</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 156, 116); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">3184</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(253, 205, 126); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">6037</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">256</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(151, 205, 126); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">18823</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(216, 224, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">11428</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(229, 228, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">10000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 205, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5470</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 191, 123); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5245</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(247, 233, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8648</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">512</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(122, 197, 125); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22068</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(176, 213, 128); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(225, 227, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">10491</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(238, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9014</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(237, 230, 131); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">9552</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(192, 217, 129); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13913</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">1024</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(115, 195, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22857</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(146, 204, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">19393</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7356</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(200, 219, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13333</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(195, 218, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13617</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(162, 208, 127); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16842</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">2048</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(108, 193, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">23703</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(129, 199, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">21333</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(164, 209, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">17297</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(152, 206, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">17777</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(122, 197, 125); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">20645</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">4096</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(108, 193, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">23703</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(122, 197, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22068</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7804</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(146, 204, 126); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">19393</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(129, 199, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">20000</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(115, 195, 124); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">21333</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">8192</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(108, 193, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">23703</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(115, 195, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22857</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(250, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7619</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(122, 197, 125); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22068</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(108, 193, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22068</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(99, 190, 123); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22857</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">16384</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(108, 193, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">23703</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(115, 195, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">22857</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7804</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(164, 209, 127); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">17297</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(115, 195, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">21333</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(147, 204, 126); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">18285</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">32768</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(172, 211, 128); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16410</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(172, 211, 128); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16410</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(204, 221, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">12800</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(171, 211, 128); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">16000</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(204, 221, 130); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">12800</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">65536</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(202, 220, 129); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13061</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(186, 215, 128); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">14883</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(202, 220, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13061</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(186, 215, 128); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">14545</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(201, 220, 129); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13061</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">131072</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(192, 217, 129); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">14222</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(194, 218, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13913</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(249, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(204, 221, 130); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">12800</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(195, 218, 129); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">13617</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(204, 221, 130); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">12800</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">262144</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(252, 196, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 197, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5039</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7032</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(248, 233, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7901</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 187, 122); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5000</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7804</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">524288</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(253, 198, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5079</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 196, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5000</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7356</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(245, 232, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">8205</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 188, 123); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5079</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(255, 235, 132); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7804</td> </tr>
<tr height="20" style="height: 15pt;"> <td align="right" class="xl71" height="20" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext; border-style: solid; border-width: 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15pt; text-decoration: none;">1048576</td> <td align="right" class="xl65" style="background: none repeat scroll 0% 0% rgb(252, 194, 124); border-color: white white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4885</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 194, 124); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4885</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(253, 235, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7272</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 234, 132); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7441</td> <td align="right" style="background: none repeat scroll 0% 0% rgb(252, 182, 121); border: 0.5pt solid white; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">4671</td> <td align="right" class="xl66" style="background: none repeat scroll 0% 0% rgb(254, 230, 131); border-color: white windowtext white white; border-style: solid; border-width: 0.5pt 1pt 0.5pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7529</td> </tr>
<tr height="21" style="height: 15.75pt;"> <td align="right" class="xl72" height="21" style="background: none repeat scroll 0% 0% rgb(220, 230, 241); border-color: white windowtext windowtext; border-style: solid; border-width: 0.5pt 1pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; height: 15.75pt; text-decoration: none;">2097152</td> <td align="right" class="xl67" style="background: none repeat scroll 0% 0% rgb(252, 197, 124); border-color: white white windowtext windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5039</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(253, 198, 124); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5079</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(253, 235, 132); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7272</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(250, 234, 132); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7619</td> <td align="right" class="xl68" style="background: none repeat scroll 0% 0% rgb(252, 187, 122); border-color: white white windowtext; border-style: solid; border-width: 0.5pt 0.5pt 1pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">5000</td> <td align="right" class="xl69" style="background: none repeat scroll 0% 0% rgb(254, 233, 131); border-color: white windowtext windowtext white; border-style: solid; border-width: 0.5pt 1pt 1pt 0.5pt; color: black; font-family: Calibri; font-size: 11pt; font-weight: 400; text-decoration: none;">7710</td> </tr>
</tbody></table>
<br />
Graph comparison only for cpblk, memcpy and CustomCopy:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMnXbi4qulDlHQXp3ZvkyJcVO7Taa4fuBOjkg2EnlipVQkcpgGFln-Gvrjn2_baidkOhK9zr1Gle0wfnfUsshagUVmDlPi2KLvmV3Epjj6nIW9gMCDP1bfm8LvnD-FrYpsfDUiGJwDr1Q/s1600/MemcpyPerf.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="276" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMnXbi4qulDlHQXp3ZvkyJcVO7Taa4fuBOjkg2EnlipVQkcpgGFln-Gvrjn2_baidkOhK9zr1Gle0wfnfUsshagUVmDlPi2KLvmV3Epjj6nIW9gMCDP1bfm8LvnD-FrYpsfDUiGJwDr1Q/s640/MemcpyPerf.png" width="640" /></a></div>
<br />
Don't be afraid about the performance drop for most of the implem... It's mostly due to cache missing and copying around different 4k pages.<br />
<br />
<h3>
Conclusion</h3>
Don't trust your .NET VM, check your code on both x86 and x64. It's interesting to see how much the same task is implemented differently inside the CLR (see Marshal.Copy vs Array.Copy vs Buffer.Copy)<br />
<br />
The most surprising result here is <b>the poor performance of cpblk IL instruction in x86 mode </b>compare to the best one in x64 which is... cpblk. So to summarize:<br />
<ul>
<li><b>On x86, you should better use a <u>memcpy </u>function</b></li>
<li><b>On x64, you should better use a <u>cpblk </u>function</b>, which is performing better from small size (twice faster than memcpy) to large size.</li>
</ul>
<b>You may wonder why the x86 version is so unoptimized? </b>This is because the x86 CLR is generating a x86 instruction that is performing a memcpy on a PER BYTE basis (rep movb for x86 folks), even if you are moving a large memory chunk of 1Mo! In comparison, a memcpy as implemented in MSVCRT is able to use SSE instructions that are able to batch copy with large 128 bits registers (with also an optimized case for not poluting CPU cache). This is the case for x64 that seems to use a correct implemented memcpy, but the x86 CLR memcpy is just poorly implemented. Please vote for this <a href="https://connect.microsoft.com/VisualStudio/feedback/details/766977/il-bytecode-method-cpblk-badly-implemented-by-x86-clr" target="_blank">bug described on Microsoft Connect</a>.<br />
<br />
One important consequence of this is when you are developping a C++/CLI and calling a memcpy from a managed function... It will end up in a cpblk copy functions... which is almost the worst case on x86 platforms... so be careful if you are dealing with this kind of issue. To avoir this, you have to force the compiler to use the function from the MSVCRTxx.dll. <br />
<br />
Of course, the memcpy is platform dependent, which would not be an option for all...<br />
<br />
Also, I didn't perform this test on a CLR 2 runtime... we could be surprised as well... There is also one thing that I should try against a pure C++ memcpy using the optimized SSE2 version that is shipped with later msvcrt.<br />
<br />
You can download the VS2010 project from <a href="http://xoofx.com/upload/BenchmarkCLI.7z">here</a>xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com7tag:blogger.com,1999:blog-1076643699683521890.post-44320545823003941652010-10-19T21:36:00.203+11:002010-12-07T23:54:21.991+11:00A new managed .NET/C# Direct3D 11 API generated from DirectX SDK headers<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjqlr9KJzUc-hZ7tzMvEmFG55-gmyM_fjGNHVNGs376NTUgnCGWkU7PSi7Jhk2HuZw1tUyEvyg8FrMHNZw9cTfmkBTJYbT3FTij6GCxkNW6Ei9Os7YAcaP-6MpdBz_pnhBDNRu4osM57U/s1600/SharpDXMiniTri.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="156" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjqlr9KJzUc-hZ7tzMvEmFG55-gmyM_fjGNHVNGs376NTUgnCGWkU7PSi7Jhk2HuZw1tUyEvyg8FrMHNZw9cTfmkBTJYbT3FTij6GCxkNW6Ei9Os7YAcaP-6MpdBz_pnhBDNRu4osM57U/s200/SharpDXMiniTri.png" width="200" /></a>I have been quite busy since the end of august, personally because I'm proud to announce the birth of my daughter! (and his older brother, is somewhat, asking a lot more attention since ;) ) and also, working hard on an exciting new project based on .NET and Direct3D.<br />
<br />
What is it? Yet Another Triangle App? Nope, this is in fact an entirely new .NET API for Direct3D11, DXGI, D3DCompiler that is fully managed without using any mixed assemblies C++/CLI but having similar performance than a true C++/CLI API (like SlimDX). But the main characteristics and most exciting thing about this new wrapper is that<b> the whole code marshal/interop is fully generated from the DirectX SDK headers, including the MSDN documentation</b>.<br />
<br />
The current key features and benefits of this approach are:<br />
<br />
<ul><li><b>API is generated from DirectX SDK headers </b><b> </b>: the mapping is able to perform "complex transformation", extracting all relevant information like enumerations, structures, interfaces, functions, macro definitions, guids from the C++ source headers. For example, the mapping process is able to generated properties for interfaces or inner group interface like the one you have in SlimDX : meaning that instead of having a "device.IASetInputLayout" you are able to write "device.InputAssembler.InputLayout = ...".</li>
<li><b>Full support of Direct3D 11, DXGI 1.0/1.1, D3DCompiler API </b>: Due to the whole auto-generated process, the actual coverage is 100%. Although, I have limited the generated code to those library but that could be extended to others API quite easily (like XAudio2, Direct2D, DirectWrite... etc.).</li>
<li><b>Pure managed .NET API </b>: assemblies are compiled with <b>AnyCpu target</b>. You can run your code on a x64 or a x86 machine with the same assemblies. </li>
<li><b>API Extensibility</b> The generated code is in C#, all the types are marked "partial" and are easily extensible to provide new helpers method. The code generator is able to hide some methods/types internally in order to use them in helper methods and to hide them from the public api. </li>
<li><b>C++/CLI Speed </b>: the framework is using a genuine way to avoid any C++/CLI while still achieving comparable performance. </li>
<li><b>Separate assemblies </b>: a core assembly containing common classes and an assembly for each subgroup API (Direct3D, DXGI, D3DCompiler)</li>
<li><b>Lightweight assemblies </b>: generated assemblies are lightweight, 300Ko in total, 70Ko compressed in an archive (similar assemblies in C++/CLI would be closer to 1Mo, one for each architecture, and depend from MSVCRT10)</li>
<li><b>API naming convention very close to SlimDX API </b>(To make it 100% equals would just require to specify the correct mapping names while generating the code)</li>
<li><b>Raw DirectX object life management</b> : No overhead of ObjectTable or <a href="http://msdn.microsoft.com/en-us/library/8bwh56xe.aspx">RCW </a>mechanism, the API is using direct native management with classic COM method "Release". Currently, instead of calling Dispose, you should call Release (and call AddRef if you are duplicating references, like in C++). I might evaluate how to safely integrate Dispose method call.<b> </b></li>
<li><b>Easily obfuscatable</b> : Due to the fact the framework is not using any mixed assemblies</li>
<li><b>DirectX SDK Documentation integrated in the .NET xml comments </b>: The whole API is also generated with the MSDN documentation. Meaning that you have exactly the same documentation for DirectX and for this API (this is working even for method parameters, remarks, enum items...etc.). Reference to other types inside the documentation are correctly linked to the .NET API. </li>
<li><b>Prototype for a partial support of the Effects11 API in full managed .NET</b>.</li>
</ul><div class="separator" style="clear: both; text-align: center;"></div>If you have been working with SlimDX, some of the features here could sound familiar and you may wonder why another .DirectX NET API while there is a great project like SlimDX? Before going further in the detail of this wrapper and how things are working in the background, I'm going to explain why this wrapper could be interesting.<br />
<br />
I'm also currently not in the position to release it for the reason that I don't want to compete with SlimDX. I want to see if SlimDX Team would be interested to work together with this system, a kind of joint-venture. There are still lots of things to do, improving the mapping, making it more reliable (the whole code here has been written in a urge since one month...) but I strongly believe that this could be a good starting point to SlimDX 2, but I might be wrong... also, SlimDX could think about another road map... So <b>this is a message to the SlimDX Team : Promit, Josh, Mike, I would be glad to hear some comments from you about this wrapper </b> (and if you want, I could send you the generated API so that you could look at it and test it!)<br />
<br />
<b>[Updated 30 November 2010]</b><br />
This wrapper is now available from <a href="http://code.google.com/p/sharpdx/">SharpDX</a>. Check this <a href="http://code4k.blogspot.com/2010/11/official-release-of-sharpdx-10.html">post</a>.<br />
<b>[/Updated]</b><br />
<br />
This post is going to be quite long, so if you are not interested by all the internals, you could jump to the sample code at the end. <br />
<br />
<a name='more'></a><h3>An attempt to a SlimDX next gen</h3><br />
First of all, is it related to 4k or 64k intros? (an usual question here, mostly question for myself :D) Well, while I'm still working to make things smaller, even in .NET, I would like to work on a demo based on .NET (but with lots of procedurally generated textures and music). I have been evaluating both XNA and SlimDX, and in September, I have even been working on a XNA like API other SlimDX / Direct3D 11 that was working great, simplifiying a lot the code, while still having benefits to use new D3D11 API (Geometry shaders, Compute Shaders...etc.). I will talk later about this "Demo" layer API.<br />
<br />
As a demo maker for tiny executable, even in .NET, I found that working with SlimDX was not the best option : even stripping the code, recompiling the SlimDX to keep only DirectX11/DXGI&co, I had a roughly 1Mo dll (one for each architecture) + a dependency to MSVRT10 which is a bit annoying. Even if I would like to work on a demo (with less size constraint), I didn't want to have a 100Ko exe and a 1Mo compressed of external dlls...<br />
<br />
Also, I read some of <a href="http://scientificninja.com/tag/slimdx">Josh's thoughts about SlimDX 2</a> : I was convinced about the need for <a href="http://scientificninja.com/blog/thoughts-on-slimdx-2-assembly-design">separated assemblies </a>and <a href="http://scientificninja.com/blog/thoughts-on-slimdx-2-object-lifetime">simplified life object management</a>. But was not convinced by the need to <a href="http://scientificninja.com/blog/thoughts-on-slimdx-2-interfaces">use "interfaces" for the new API </a>and not really happy about still having some platform specific mixed-assemblies in order to support correctly 32/64 bit architecture (with a simple delay loading).<br />
<br />
What is supposed to address SlimDX 2 over SlimDX? <br />
<ul><li>Making object life management closer to the real thing (no Dispose but raw Release instead) </li>
<li>Multiple assemblies</li>
<li>Working on the API more with C# than in C++/CLI</li>
<li>Support automatic platform architecture switching (running transparently an executable on a x86 and x64 machine without recompiling anything).</li>
</ul>Recall that I was slightly working around August on parsing the SDK headers based on <a href="http://www.boost.org/doc/libs/1_43_0/libs/wave/index.html">Boost::Wave V2.0</a>. My concern was that I have developed a SlimDX like interface in C++ for Ergon demo, but I found the process to be very laborious, although very straightforward, while staying in the same language as DirectX... Thinking more about it, and because I wanted to do more work in 3D and C# (damn it, this language is SOOO cool and powerful compared to C++)... I found that It would be a great opportunity to see if it's not possible to extract enough information from the SDK headers in order to generate a Direct3D 11 .NET/C# API.<br />
<br />
And everything has been surprisingly very fast : extraction of all the code information from the SDK C++ headers file was in fact quite easy to code, in few days... and generating the code was quite easy (I have to admit that I have a strong experience in this kind of process, and did similar work, around ten years ago, in Java, delivering an innovative Java/COM bridge layer for the company I was working at that time, much safer than Sun Java/COM layer that was buggy and much more powerfull, supporting early binding, inheritance, documentation... etc).<br />
<br />
In fact, with this generating process, I have been able to address almost all the issue that were expected to be solved in SlimDX 2, and moreover, It's going a bit further because the process is automated and It's supporting the platform x86/x64 without requiring any mixed assemblies.<br />
<br />
In the following sections, I'm going to deeply explain the architecture, features, internals and mapping rules used to generate this new .Net wrapper (which has currently the "SharpDX" code name).<br />
<br />
<h2>Overview </h2><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic3bBMqTiNAnwfnehAqbEkxVQaiGEA31A5eJKcfYmqEuJhe1Zv2-0LUas9OxH00i3FXysl-Jwat9MOeg7HdeTQI5fvmiyA8na8q8eqPLLxpYRpNThPQY1mMqWOoil-Zqrev8E1gYb4ABc/s1600/SharpDXGeneral.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="520" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic3bBMqTiNAnwfnehAqbEkxVQaiGEA31A5eJKcfYmqEuJhe1Zv2-0LUas9OxH00i3FXysl-Jwat9MOeg7HdeTQI5fvmiyA8na8q8eqPLLxpYRpNThPQY1mMqWOoil-Zqrev8E1gYb4ABc/s640/SharpDXGeneral.png" width="640" /></a></div>In order to generate Managed .NET API for DirectX from the SDK headers, the process is composed of 3 main steps:<br />
<ol><li><a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165#HeadersToXIDL">Convert from the DirectX SDK C++ Headers to an intermediate format called "XIDL" which is a mix of XML and "IDL"</a>. This first part is responsible to reverse engineering the headers, extract back all existing and useful information (more on the following section), and produce a kind of IDL (Intermediate Definition Language). In fact, If I had access to the IDL used internally at Microsoft, It wouldn't have been necessary to write this whole part, but sadly, the DirectX 11 IDL is not available, although you can clearly verify from the D3D11.h that this file is generated from an IDL. This module is also responsible to access MSDN website and crawl the needed documentation, and associate it with all the languages elements (structures, structures fields, enums, enum items, interfaces, interfaces methods, method parameters...etc.). Once a documentation has been retrieved, It's stored on the disk and is not retrieved next time the conversion process is re-runned.</li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165#XIDLToCSharp">Convert from the XIDL file to several C# files</a>. This part is responsible to perform from a set of mapping rules a translation of C++ definition to C# definition. The mapping is as complex as identifying which include would map to assembly/namespace, which type could be moved to an assembly/namespace, how to rename the types,functions, fields, parameters, how to add missing information from the XIDL file...etc. The current mapping rules are express in less then 600 lines of C# code... There is also a trick here not described in the picture. This process is also generating a small interop assembly which is only used at compile time, dynamically generated at runtime and responsible for filling the gap between what is possible in C# and what you can do in C++/CLI (there are lots of small usefull IL bytecode instructions generated in C++/CLI that are not accessible from C#, this assembly is here for that....more on this in the Convert to XIDL section).</li>
<li><a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165#CSharpToFiles">Integrate the generated files in several Visual Studio projects and a global solution</a>. Each project is generating an assembly. It is where you can add custom code that could not be generated (like Vector3 math functions, or general framework objects like a ComObject). The generated code is also fully marked with "partial" class, one of the cool things of C# : you can have multiple files contributing to the same class declaration... making things easy to have generated code on the side of custom hand made code. </li>
</ol><br />
<a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165" name="HeadersToXIDL"></a><br />
<h2>Revert DirectX IDL from headers</h2><br />
Unfortunately, I have not found a workable C preprocessor written in .NET, and this part has been a bit laborious to make it work. The good thing is that I have found <a href="http://www.boost.org/doc/libs/1_43_0/libs/wave/index.html">Boost Wave 2.0</a> in C++. The bad thing is that this library, written in a heavy boost-STL-templatizer philosophy was really hard to manage to work under a C++/CLI DLL. Well, the principle was to embed Boost Wave in a managed DLL, in order to use it from C#... after several attempts, I was not able to build it with C++/CLI .NET 4.0. So I ended up in a small dll COM wrapper around BoostWave, and a thin wrapper in .NET calling this dll. Compiling Boost-Wave was also sometimes a nightmare : I tried to implement my own provider of stream for Wave... but dealing with a linker error that was freezing VS2010 for 5s to display the error (several Ko of a single template cascaded error)... I have found somewhere on the Wave release that It was in fact not supported... but wow, templates are supposed to make life easier... but the way It is used gives a really bad feeling... (and I'm not a beginner in C++ template...)<br />
<br />
Anyway, after succeeding to wrap BoostWave API, I had a bunch of tokens to process. I started to wrote a handwritten C/C++ parser, which is targeted to read well-formed DirectX headers and nothing else. It was quite tricky sometimes, the code is far from being failsafe, but I succeed to parse correctly most of the DirectX headers. During the mapping to C#, I was able to find a couple of errors in the parser that were easy to fix.<br />
<br />
In the end, this parser is able to extract from the headers:<br />
<ul><li>Enumerations, Structures, Interfaces, Functions, Typedefs</li>
<li>Macros definitions</li>
<li>GUIDs</li>
<li>Include dependency</li>
</ul>The whole data is stored in a C# model that is marshaled in XML using WCF (DataMember, DataContract), which make the code really easy to write, not much intrusive and you can serialize and deserialize to XML. For example, a CppType is defined like this:<br />
<br />
<pre class="brush: csharp" name="code">//
using System.Runtime.Serialization;
using System.Text;
namespace SharpDX.Tools.XIDL
{
[DataContract]
public class CppType : CppElement
{
[DataMember(Order=0)]
public string Type { get; set;}
[DataMember(Order=1)]
public string Specifier { get; set; }
[DataMember(Order=2)]
public bool Const { get; set; }
[DataMember(Order = 3)]
public bool IsArray { get; set; }
[DataMember(Order=4)]
public string ArrayDimension { get; set; }
</pre><br />
The model is really lightweight, no fancy methods and easy to navigate in.<br />
<br />
The process is also responsible to get documentation for each C++ items (enumerations, structures, interfaces, functions). The documentation is requested to MSDN while generating all the types. That was also a bit tricky to parse, but in the end, the class is very small (less than 200 lines of C# code). Downloaded documentation is stored on the disk and is used for later re-generation of the parsing. <br />
<br />
The generated XML model is taking around 1.7Mo for DXGI, D3D11, D3DX11, D3DCompiler includes and looks like this:<interfaces></interfaces><br />
<br />
<pre style="font-family: consolas;"><span style="color: blue;"> <</span><span style="color: #a31515;">Interfaces</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">CppInterface</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Name</span><span style="color: blue;">></span>ID3D11DeviceChild<span style="color: blue;"><span style="color: #a31515;">Name</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Description</span><span style="color: blue;">></span>A device-child interface accesses data used by a device.<span style="color: blue;"><span style="color: #a31515;">Description</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Remarks</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Parent</span><span style="color: blue;">></span>IUnknown<span style="color: blue;"><span style="color: #a31515;">Parent</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Methods</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">CppMethod</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Name</span><span style="color: blue;">></span>GetDevice<span style="color: blue;"><span style="color: #a31515;">Name</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Description</span><span style="color: blue;">></span>Get a pointer to the device that created this interface.<span style="color: blue;"><span style="color: #a31515;">Description</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Remarks</span><span style="color: blue;">></span>Any returned interfaces will have their reference count incremented by one, so be sure to call ::release() on the returned pointer(s) before they are freed or else you will have a memory leak.<span style="color: blue;"><span style="color: #a31515;">Remarks</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">ReturnType</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Name</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Description</span><span style="color: blue;">></span>voidReturns nothing.<span style="color: blue;"><span style="color: #a31515;">Description</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Remarks</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Type</span><span style="color: blue;">></span>void<span style="color: blue;"><span style="color: #a31515;">Type</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Specifier</span><span style="color: blue;">><span style="color: #a31515;">Specifier</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Const</span><span style="color: blue;">></span>false<span style="color: blue;"><span style="color: #a31515;">Const</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">IsArray</span><span style="color: blue;">></span>false<span style="color: blue;"><span style="color: #a31515;">IsArray</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">ArrayDimension</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <span style="color: #a31515;">ReturnType</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">CallingConvention</span><span style="color: blue;">></span>StdCall<span style="color: blue;"><span style="color: #a31515;">CallingConvention</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Offset</span><span style="color: blue;">></span>3<span style="color: blue;"><span style="color: #a31515;">Offset</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Parameters</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">CppParameter</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Name</span><span style="color: blue;">></span>ppDevice<span style="color: blue;"><span style="color: #a31515;">Name</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Description</span><span style="color: blue;">></span>Address of a pointer to a device (see {{ID3D11Device}}).<span style="color: blue;"><span style="color: #a31515;">Description</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Remarks</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Type</span><span style="color: blue;">></span>ID3D11Device<span style="color: blue;"><span style="color: #a31515;">Type</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Specifier</span><span style="color: blue;">></span>**<span style="color: blue;"><span style="color: #a31515;">Specifier</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Const</span><span style="color: blue;">></span>false<span style="color: blue;"><span style="color: #a31515;">Const</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">IsArray</span><span style="color: blue;">></span>false<span style="color: blue;"><span style="color: #a31515;">IsArray</span><span style="color: blue;">></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">ArrayDimension</span><span style="color: blue;"> </span><span style="color: red;">i:nil</span><span style="color: blue;">=</span>"<span style="color: blue;">true</span>"<span style="color: blue;"> /></span>
<span style="color: blue;"> <</span><span style="color: #a31515;">Attribute</span><span style="color: blue;">></span>Out<span style="color: blue;"><span style="color: #a31515;">Attribute</span><span style="color: blue;">></span>
<span style="color: blue;"> <span style="color: #a31515;">CppParameter</span><span style="color: blue;">></span>
<span style="color: blue;"> <span style="color: #a31515;">Parameters</span><span style="color: blue;">></span>
<span style="color: blue;"> <span style="color: #a31515;">CppMethod</span><span style="color: blue;">></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></pre><br />
One of the most important thing in the DirectX headers that are required <b>to develop a reliable code generator is the presence of C+ windows specific attributes</b> : all the methods are prefix by macros __out __in __out_opt , __out_buffer... etc. All those attributes are similar to C# attributes and are explaining how to interpret the parameter. If you take the previous code, there is a method GetDevice that is returning a ID3D11Device through a [out] parameter. The [Out] parameter is extremely important here, as we know exactly how to use it. Same thing when you have a pointer which is in fact a buffer : with the attributes, you know that this is an array of elements behind the pointer...<br />
<br />
Although, I have discovered that some functions/methods sometimes are lacking some attributes.... but hopefully, the next process (the mapping from XIDL to C#) is able to add missing information like this.<br />
<br />
<br />
As I said, the current implementation is far from being failsafe and would probably require more testing on other headers files. At least, the process is correctly working on a subset of the DirectX headers.<br />
<br />
<a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165" name="XIDLToCSharp"></a><br />
<h2>Generate C# from IDL </h2><br />
This part of the process has been a lot more time consuming. I started with enums, which were quite straightforward to manage. Structures were asking a bit more work, as there is some need for some custom marshalling for some structures that cannot marshal easily... Then interfaces methods were the most difficult part, correctly handling all parameters case was not easy...<br />
<br />
The process of generating the C# code is done in 3 steps:<br />
<ol><li><b>Reading XIDL model and prepare the model for mapping</b>: remove types, add information to some methods. </li>
<li><b>Generate a C# model with the XIDL model and a set of mapping rules</b></li>
<li><b>Generate C# files from the C# model</b>. I have used <a href="http://www.olegsych.com/2007/12/text-template-transformation-toolkit/">T4 "Text Template Transformation Toolkit" </a>engine as a text templatizer, which is part of VS2010 and is really easy to use, integrated in VS2010 with a third party syntax highlighting plugin. </li>
</ol>This step is also responsible to <b>generate an interop assembly </b>which is emiting directly some .NET IL bytecodes through the System.Reflection.Emit. This interop assembly is the trick to avoid the usage of a C++/CLI mixed assembly<br />
<br />
<h3>Preamble) How to avoid the usage of C++/CLI in C#</h3><br />
If you look at some generated C++/CLI code with Reflector, you will see that most of the code is in fact a pure IL bytecode, even when there is a call to a native function or native methods...<br />
<br />
The trick here is that there are a couple of IL instructions that are used internally by C# but not exposed to the language. <br />
<br />
1) The <b>instruction "<a href="http://msdn.microsoft.com/en-us/library/system.reflection.emit.opcodes.calli.aspx">calli</a>" </b><br />
<br />
This instruction is responsible to call directly an unmanaged function, without going through the pinvoke/interop layer (in fact, pinvoke is calling in the end "calli", but is performing a much more complex marshaling of the parameters, structures...)<br />
<br />
What I need was a way to call an umanaged function/methods without going through the pinvoke layer, and "calli" is exactly here for this. Now, suppose that we could generate a small assembly at compile time and at runtime that would be responsible for handling those calli function, we would not have to use anymore C++/CLI for this.<br />
<br />
For example, suppose that I want to call a C++ method of an interface which takes an integer as a parameter, something like :<br />
<pre class="brush: csharp" name="code">interface IDevice : IUnknown {
void Draw(int count);
}
</pre>I only need a function in C# that is able to directly call this method, without going the pinvoke layer, with a pointer to the C++ IDevice object and the offset of the method in the vtbl (offset will be expressed in bytes, for a x86 architecture here) :<br />
<pre class="brush: csharp" name="code">class Interop {
public static unsafe void CalliVoid(void* thisObject, int vtblOffset, int arg0);
}
// A call to IDevice
void* ptrToIDevice = ...;
// A Call to the method Draw, number 3 in the vtbl order (starting at 0 to 2 for IUnknown methods)
Interop.CalliVoid(ptrToIDevice, /* 3 * sizeof(void* in x86) */ 3 * 4 , /* count */4 );
</pre><br />
The IL bytecode content of this method for a x64 architecture would be typically in C++/CLI like this:<br />
<pre class="brush: csharp" name="code">.method public hidebysig static void CalliVoid(void* arg0, int32 arg1, int32 arg2) cil managed
{
.maxstack 4
L_0000: ldarg.0 // Load (0) this arg (1st parameter for native method)
L_0001: ldarg.2 // Load (1) count arg
L_0002: ldarg.1 // Offset in vtbl
L_0003: conv.i // Convert to native int
L_0004: dup //
L_0005: add // Offset = offset * 2 (only for x64 architecture)
L_0006: ldarg.0 //
L_0007: ldind.i // Load vtbl poointer
L_0008: add // pVtbl = pVtbl + offset
L_0009: ldind.i // load function from the vtbl fointer
L_000a: calli method unmanaged stdcall void *(void*, int32)
L_000f: ret
}
</pre><br />
This kind of code will be automatically inlined by the JIT (which is, from SCCLI/Rotor sourcecode, inlining functions that are taking less than 25 bytes of bytecode).<br />
<br />
If you look at a C++/CLI assembly, you will see lots of "calli" instructions.<br />
<br />
So in the end, how this trick is used? Because the generator knows all the methods from all the interfaces, it is able to generate a set of all possible calling conventions to unmanaged object. In fact, the XIDLToCSharp generator is responsible to generate an assembly containing all the interop methods (around 66 methods using Calli) :<br />
<pre class="brush: csharp" name="code">public class Interop
{
private Interop();
public static unsafe float CalliFloat(void* arg0, int arg1, void* arg2);
public static unsafe int CalliInt(void* arg0, int arg1);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2);
public static unsafe int CalliInt(void* arg0, int arg1, long arg2);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, int arg3);
public static unsafe int CalliInt(void* arg0, int arg1, long arg2, int arg3);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, int arg3);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, void* arg3);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, void* arg3);
public static unsafe int CalliInt(void* arg0, int arg1, IntPtr arg2, void* arg3);
public static unsafe int CalliInt(void* arg0, int arg1, IntPtr arg2, int arg3);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, void* arg3, int arg4);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, void* arg3, void* arg4);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, int arg3, void* arg4);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, int arg3, void* arg4);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, void* arg3, void* arg4);
public static unsafe int CalliInt(void* arg0, int arg1, IntPtr arg2, void* arg3, void* arg4);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, void* arg3, int arg4);
public static unsafe int CalliInt(void* arg0, int arg1, int arg2, int arg3, void* arg4, void* arg5);
public static unsafe int CalliInt(void* arg0, int arg1, void* arg2, void* arg3, int arg4, int arg5);
//
// ...[stripping Calli x methods here]...
//
public static unsafe void CalliVoid(void* arg0, int arg1, int arg2, void* arg3, void* arg4, int arg5, int arg6, void* arg7);
public static unsafe void CalliVoid(void* arg0, int arg1, void* arg2, float arg3, float arg4, float arg5, float arg6, void* arg7);
public static unsafe void CalliVoid(void* arg0, int arg1, int arg2, void* arg3, void* arg4, int arg5, int arg6, void* arg7, void* arg8);
public static unsafe void CalliVoid(void* arg0, int arg1, void* arg2, int arg3, int arg4, int arg5, int arg6, void* arg7, int arg8, void* arg9);
public static unsafe void* Read<T>(void* pSrc, ref T data) where T: struct;
public static unsafe void* Read<T>(void* pSrc, T[] data, int offset, int count) where T: struct;
public static unsafe void* Write<T>(void* pDest, ref T data) where T: struct;
public static unsafe void* Write<T>(void* pDest, T[] data, int offset, int count) where T: struct;
public static void memcpy(void* pDest, void* pSrc, int Count);
}</pre><br />
This assembly is used at compile time but is not distributed at runtime. Instead, this assembly is dynamically generated at runtime in order to support difference in bytecode between x86 and x64 (in the calli example, we need to multiply by 2 the offset into the vtbl table, because the sizeof of a pointer in x64 is 8 bytes).<br />
<br />
2) The <b>instruction "<a href="http://msdn.microsoft.com/en-us/library/system.reflection.emit.opcodes.sizeof.aspx">sizeof</a>" for generic</b><br />
<br />
Although the Calli is the real trick that makes it possible to have a managed way to call unmanaged method without using pinvoke, I have found a couple of other IL bytecode that is necessary to have the same features than in C++/CLI.<br />
<br />
The other one is sizeof for generic. In C#, we know that there is a sizeof, but while trying to replicate the DataStream class from SlimDX in pure C#, I was not able to write this kind code :<br />
<pre class="brush: csharp" name="code">public class DataStream
{
// Unmarshal a struct from a memory location
public T Read<T>() where T: struct {
T myStruct = default(T);
memcpy(&mystruct, &m_buffer, sizeof(T));
return myStruct;
}
}
</pre><br />
In fact, under C#, the sizeof is not working for a generic, even if you specify that the generic is a struct. Because C# cannot constraint the struct to contains only blittable fields (I mean, It could, but It doesn't try to do it), they don't allow to take the size of a generic struct... that was annoying, but because with pure IL instruction, It's working well and I was already generating the Interop assembly, I was free to add whatever methods with custom bytecode to fill the gap...<br />
<br />
In the end, the interop code to read a generic struct from a memory location looks like this :<br />
<pre class="brush: csharp" name="code">// This method is reading a T struct from pSrc and returning the address : pSrc + sizeof(T)
.method public hidebysig static void* Read<valuetype .ctor T>(void* pSrc, !!T& data) cil managed
{
.maxstack 3
.locals init (
[0] int32 num,
[1] !!T* pinned localPtr)
L_0000: ldarg.1
L_0001: stloc.1
L_0002: ldloc.1
L_0003: ldarg.0
L_0004: sizeof !!T
L_000a: conv.i4
L_000b: stloc.0
L_000c: ldloc.0
L_000d: unaligned 1 // Mandatory for x64 architecture
L_0010: nop
L_0011: nop
L_0012: nop
L_0013: cpblk // Memcpy
L_0015: ldloc.0
L_0016: conv.i
L_0017: ldarg.0
L_0018: add
L_0019: ret
}
</pre><br />
3) The <b>instruction "<a href="http://msdn.microsoft.com/en-us/library/system.reflection.emit.opcodes.cpblk.aspx">cpblk</a>", memcpy in IL</b><br />
<br />
In the previous function, you can see the use of "cpblk" bytecode instruction. In fact, when you are looking at a C++/CLI method using a memcpy, It will not use the memcpy from the C CRT but directly the IL instruction performing the same task. This IL instruction is faster than using anykind of interop, so I made it available to C# through the Interop assembly<br />
<br />
<h3>I) Prepare XIDL model for mapping</h3><br />
So the 1st step in the XIDLToCSharp process is to prepare the XIDL model to be more mapping friendly. This step is essentially responsible to:<br />
<ul><li><b>Add missing C++ attributes </b>(In, InOut, Buffer) information to some method's parameter</li>
<li><b>Replace the type of some method parameters </b>: for example in DirectX, there are lots of parameter that are taking a flags, which is in fact an already declared enum... but for some unknown reason, they are declaring the method with an "int" instead of using the enum... </li>
<li><b>Remove some types</b>. For example, the D3D_PRIMITIVE_TOPOLOGY is holding a bunch of D3D11 and D3D10 enum, duplicating D3D_PRIMITIVE enums... So I'm removing them. </li>
<li><b>Add some tag directly on the XIDL</b> model in order to ease the next mapping process : those tags are for example used for tagging the C# visibility of the method, or forcing a method to not be interpreted as a "property")</li>
</ul><pre class="brush: csharp" name="code">// Read the XIDL model
CppIncludeGroup group = CppIncludeGroup.Read("directx_idl.xml");
group.Modify<CppParameter>("^D3DX11.*?::pDefines", Modifiers.ParameterAttribute(CppAttribute.In | CppAttribute.Buffer | CppAttribute.Optional));
// Modify device Flags for D3D11CreateDevice to use D3D11_CREATE_DEVICE_FLAG
group.Modify<CppParameter>("^D3D11CreateDevice.*?::Flags$", Modifiers.Type("D3D11_CREATE_DEVICE_FLAG"));
// ppFactory on CreateDXGIFactory.* should be Attribute.Out
group.Modify<CppParameter>("^CreateDXGIFactory.*?::ppFactory$", Modifiers.ParameterAttribute(CppAttribute.Out));
// pDefines is an array of Macro (and not just In)
group.Modify<CppParameter>("^D3DCompile::pDefines", Modifiers.ParameterAttribute(CppAttribute.In | CppAttribute.Buffer | CppAttribute.Optional));
group.Modify<CppParameter>("^D3DPreprocess::pDefines", Modifiers.ParameterAttribute(CppAttribute.In | CppAttribute.Buffer | CppAttribute.Optional));
// SwapChain description is mandatory In and not optional
group.Modify<CppParameter>("^D3D11CreateDeviceAndSwapChain::pSwapChainDesc", Modifiers.ParameterAttribute(CppAttribute.In));
// Remove all enums ending with _FORCE_DWORD, FORCE_UINT
group.Modify<CppEnumItem>("^.*_FORCE_DWORD$", Modifiers.Remove);
group.Modify<CppEnumItem>("^.*_FORCE_UINT$", Modifiers.Remove);
</pre><br />
You can see that the pre-mapping (and the mapping) is using intensively regular expression for matching names, which is a very convenient way to perform some kind of XPATH request with Regex expressions.<br />
<br />
<h3>II) Generate C# model from XIDL and mapping rules</h3><br />
This process is taking the pre-process XIDL and is generating a C# model (a subset of the C# model in memory), adding mapping information and preparing things to make it easier to use it from the T4 templatizer engine.<br />
<br />
In order to generate the C# model from DirectX, the generator needs a couple of mapping rules. <br />
<br />
1) <b>Mapping an include to an assembly / namespace</b><br />
<br />
This rules is defining a default dispatching of types to assembly / namespace. It will associate source headers include (the name of the .h, without the extension).<br />
<pre class="brush: csharp" name="code">// Namespace mapping
// Map dxgi include to assembly SharpDX.DXGI, namespace SharpDX.DXGI
gen.MapIncludeToNamespace("dxgi", "SharpDX.DXGI");
gen.MapIncludeToNamespace("dxgiformat", "SharpDX.DXGI");
gen.MapIncludeToNamespace("dxgitype", "SharpDX.DXGI");
// Map D3DCommon include to assembly SharpDX, namespace SharpDX.Direct3D
gen.MapIncludeToNamespace("d3dcommon", "SharpDX.Direct3D", "SharpDX");
gen.MapIncludeToNamespace("d3d11", "SharpDX.Direct3D11");
gen.MapIncludeToNamespace("d3dx11", "SharpDX.Direct3D11");
gen.MapIncludeToNamespace("d3dx11core", "SharpDX.Direct3D11");
gen.MapIncludeToNamespace("d3dx11tex", "SharpDX.Direct3D11");
gen.MapIncludeToNamespace("d3dx11async", "SharpDX.Direct3D11");
gen.MapIncludeToNamespace("d3d11shader", "SharpDX.D3DCompiler");
gen.MapIncludeToNamespace("d3dcompiler", "SharpDX.D3DCompiler");
</pre><br />
2) <b>Mapping a particular type to an assembly / namespace</b><br />
<br />
It is also necessary to override the default include to assembly/namespace dispatching for some particular types. This rules is doing this.<br />
<pre class="brush: csharp" name="code">gen.MapTypeToNamespace("^D3D_PRIMITIVE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_CBUFFER_TYPE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_RESOURCE_RETURN_TYPE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_CBUFFER_FLAGS$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_INPUT_TYPE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_VARIABLE_CLASS$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_VARIABLE_FLAG$S", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_VARIABLE_TYPE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_TESSELLATOR_DOMAIN$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_TESSELLATOR_PARTITIONING$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_TESSELLATOR_OUTPUT_PRIMITIVE$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_SHADER_INPUT_FLAGS$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_NAME$", "SharpDX.D3DCompiler");
gen.MapTypeToNamespace("^D3D_REGISTER_COMPONENT_TYPE$", "SharpDX.D3DCompiler");
</pre><br />
The previous code is instructing the generator to move some D3D types to the SharpDX.D3DCompiler namespace (and assembly). Those types are in fact more related to Shader reflection and are associated with the D3DCompiler assembly (I took the same design choice from SlimDX, although we could think about another mapping).<br />
<br />
3) <b>Mapping a C++ type to a custom C# type</b><br />
<br />
It is sometimes necessary to map a C++ type to a non generated C# type. For example, there is the C++ "RECT" structure which is not stritcly equivalent to the System.Drawing.Rectangle (the RECT struct is using the Left,Top,Right,Bottom fields instead of Left,Top,Width,Height for System.Drawing.Rectangle). This mapping is able to define a custom mapping. The SharpDX.Rectangle is not generated by the generator but is defined in the SharpDX assembly project (last part).<br />
<pre class="brush: csharp" name="code">var rectType = new CSharpStruct();
rectType.Name = "SharpDX.Rectangle";
rectType.SizeOf = 4*4;
gen.MapCppTypeToCSharpType("RECT", rectType); //"SharpDX.Rectangle", 4 * 4, false, true);
</pre><br />
4) <b>Mapping a C++ name to a C# name</b><br />
The renaming rules are quite rich. The XIDLToCSharp provides a default renaming mechanism that respect the CamelCase convention, but there are some exceptions that need to be addressed. For example:<br />
<pre class="brush: csharp" name="code">// Rename DXGI_MODE_ROTATION to DisplayModeRotation
gen.RenameType(@"^DXGI_MODE_ROTATION$","DisplayModeRotation");
gen.RenameType(@"^DXGI_MODE_SCALING$", "DisplayModeScaling");
gen.RenameType(@"^DXGI_MODE_SCANLINE_ORDER$", "DisplayModeScanlineOrder");
// Use regular expression to take the part of some names...
gen.RenameType(@"^D3D_SVC_(.*)", "$1");
gen.RenameType(@"^D3D_SVF_(.*)", "$1");
gen.RenameType(@"^D3D_SVT_(.*)", "$1");
gen.RenameType(@"^D3D_SIF_(.*)", "$1");
gen.RenameType(@"^D3D_SIT_(.*)", "$1");
gen.RenameType(@"^D3D_CT_(.*)", "$1");
</pre><br />
For structures and enums that are using the "_" underscore to separate name subpart, you can let XIDLToCSharp rename correctly each subpart, while still being able to specify how a subpart can be rename:<br />
<pre class="brush: csharp" name="code">// Expand sub part between underscore
gen.RenameTypePart("^DESC$", "Description");
gen.RenameTypePart("^CBUFFER$", "ConstantBuffer");
gen.RenameTypePart("^TBUFFER$", "TextureBuffer");
gen.RenameTypePart("^BUFFEREX$", "ExtendedBuffer");
gen.RenameTypePart("^FUNC$", "Function");
gen.RenameTypePart("^FLAG$", "Flags");
gen.RenameTypePart("^SRV$", "ShaderResourceView");
gen.RenameTypePart("^DSV$", "DepthStencilView");
gen.RenameTypePart("^RTV$", "RenderTargetView");
gen.RenameTypePart("^UAV$", "UnorderedAccessView");
gen.RenameTypePart("^TEXTURE1D$", "Texture1D");
gen.RenameTypePart("^TEXTURE2D$", "Texture2D");
gen.RenameTypePart("^TEXTURE3D$", "Texture3D");
</pre><br />
With this rules, for example with a struct named as "BLABLA_DESC", the DESC part will be expand to "Description", resulting in the C# name "BlablaDescription".<br />
<br />
5) <b>Change Field type mapping in C#</b><br />
<br />
Again, there are lots of enums in DirectX that are not used in the structures. For example, if you take the D3D11_BUFFER_DESC, all enums are declared as int instead of using their respective enums. <br />
<br />
This mapping rules is responsible to change the destination type for a field:<br />
<pre class="brush: csharp" name="code">gen.ChangeStructFieldTypeToNative("D3D11_BUFFER_DESC", "BindFlags", "D3D11_BIND_FLAG");
gen.ChangeStructFieldTypeToNative("D3D11_BUFFER_DESC", "CPUAccessFlags", "D3D11_CPU_ACCESS_FLAG");
gen.ChangeStructFieldTypeToNative("D3D11_BUFFER_DESC", "MiscFlags", "D3D11_RESOURCE_MISC_FLAG");
</pre><br />
6) <b>Generate enums from C++ macros, improving enums</b><br />
<br />
Again, DirectX SDK is not consistent with enums. Sometimes there are some enums that are in fact defined with some macro definition, which makes intellisense experience inexistent...<br />
<br />
XIDLToCSharp is able to create an enum from a set of macros definitions<br />
<pre class="brush: csharp" name="code">// Create enums from macro definitions
// Create the D3DCOMPILE_SHADER_FLAGS C++ type from the D3DCOMPILE_.* macros
gen.CreateEnumFromMacros(@"^D3DCOMPILE_[^E][^F].*", "D3DCOMPILE_SHADER_FLAGS");
gen.CreateEnumFromMacros(@"^D3DCOMPILE_EFFECT_.*", "D3DCOMPILE_EFFECT_FLAGS");
gen.CreateEnumFromMacros(@"^D3D_DISASM_.*", "D3DCOMPILE_DISASM_FLAGS");
</pre><br />
There are also some tiny things to adjust to existing enums, like adding a "None=0" enum item for some flags. <br />
<br />
7) <b>Move interface methods to inner interfaces in C#</b><br />
<br />
If you have been using Direct3D 11, you have notice that all methods for each stages are prefix with the stage abbreviation, making for example the <a href="http://msdn.microsoft.com/en-us/library/ff476385%28v=VS.85%29.aspx">ID3D11DeviceContext interface</a> quite ugly to use, ending in some code like this:<br />
<pre class="brush: csharp" name="code">deviceContext.IASetInputLayout(inputlayout);
</pre><br />
SlimDX did something really nice : they have created for each pipeline stage (IA for InputAssembler, VS for VertexShader) a property accessor to an interface that is exposing the method of this stage, resulting in an improved readability and a much better intellisense experience.<br />
<pre class="brush: csharp" name="code">deviceContext.InputAssembler.InputLayout = inputlayout;
</pre><br />
In the XIDL2CSharp, there is a rules to handle such a case, and is simple as writing this:<br />
<pre class="brush: csharp" name="code">// Map all IA* methods to the internal interface InputAssemblerStage with the acessor property InputAssembler, using the method name $1 (extract from the regexp)
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::IA(.*)", "InputAssemblerStage", "InputAssembler", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::VS(.*)", "VertexShaderStage", "VertexShader", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::PS(.*)", "PixelShaderStage", "PixelShader", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::GS(.*)", "GeometryShaderStage", "GeometryShader", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::SO(.*)", "StreamOutputStage", "StreamOutput", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::DS(.*)", "DomainShaderStage", "DomainShader", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::HS(.*)", "HullShaderStage", "HullShader", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::RS(.*)", "RasterizerStage", "Rasterizer", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::OM(.*)", "OutputMergerStage", "OutputMerger", "$1");
gen.MoveMethodsToInnerInterface("ID3D11DeviceContext::CS(.*)", "ComputeShaderStage", "ComputeShader", "$1");
</pre><br />
8) <b>Dispatch method to function group</b><br />
<br />
DirectX C++ functions are mapped to a set of function group and an associated DLL. For example, it is possible to specify that all D3D11.* methods will map to a class D3D11 containing all the associated methods.<br />
<pre class="brush: csharp" name="code">// Function group
var d3dCommonFunctionGroup = gen.CreateFunctionGroup("SharpDX", "SharpDX.Direct3D", "D3DCommon");
var dxgiFunctionGroup = gen.CreateFunctionGroup("SharpDX.DXGI", "SharpDX.DXGI", "DXGI");
var d3dFunctionGroup = gen.CreateFunctionGroup("SharpDX.D3DCompiler", "SharpDX.D3DCompiler", "D3D");
var d3d11FunctionGroup = gen.CreateFunctionGroup("SharpDX.Direct3D11", "SharpDX.Direct3D11", "D3D11");
var d3dx11FunctionGroup = gen.CreateFunctionGroup("SharpDX.Direct3D11", "SharpDX.Direct3D11", "D3DX11");
// Map All D3D11 functions to D3D11 Function Group
gen.MapFunctionToFunctionGroup(@"^D3D11.*", "d3d11.dll", d3d11FunctionGroup);
// Map All D3DX11 functions to D3DX11 Function Group
gen.MapFunctionToFunctionGroup(@"^D3DX11.*", group.Find<cppmacrodefinition>("D3DX11_DLL_A").FirstOrDefault().StripStringValue, d3dx11FunctionGroup);
// Map All D3D11 functions to D3D11 Function Group
string d3dCompilerDll =
group.Find<cppmacrodefinition>("D3DCOMPILER_DLL_A").FirstOrDefault().StripStringValue;
gen.MapFunctionToFunctionGroup(@"^D3DCreateBlob$", d3dCompilerDll, d3dCommonFunctionGroup);
</pre><br />
If a DLL has a versionned name (like for D3DXX_xx.dll or D3DCompiler_xx.dll), we are directly retreiving the dll name from a macro!<br />
<br />
<a href="http://www.blogger.com/post-edit.g?blogID=1076643699683521890&postID=4432054582300394165" name="CSharpToFiles"></a><br />
<h2>Generate C# code from C# model and adding custom classes</h2><br />
Once an internal C# model is built, we are calling the T4 text template toolkit engine for each group of types : Enumerations, Structures, Interfaces, Functions. Those classes are then integrated in several VS project, with some custom code added and some non generated core classes.<br />
<br />
<h3>The generated C# interop code</h3><br />
Meaning that for each assembly, each namespace, there will be an Enumerations.cs, Structures.cs, Interfaces.cs and Functions.cs files generated.<br />
<br />
For each types, there is a custom mapping done:<br />
<ul><li><b>For enums, the mapping is straightforward</b>, resulting in an almost one-to-one mapping</li>
<li><b>For structures, the mapping is quite straightforward</b>, resulting in an almost one-to-one mapping for most of the types. Although there are a couple of case where the mapping need to generate some marshalling code, essentially when there is a bool in the struct, or when there is a string pointer, or a fixed array of struct inside a struct.</li>
</ul>For example, one of the most complex mapping for a structure is generated like this:<br />
<br />
<pre class="brush: csharp" name="code">/// <summary>
/// Describes the blend state.
/// </summary>
/// <remarks>
/// These are the default values for blend state.StateDefault ValueAlphaToCoverageEnableFALSEIndependentBlendEnableFALSERenderTarget[0].BlendEnableFALSERenderTarget[0].SrcBlendD3D11_BLEND_ONERenderTarget[0].DestBlendD3D11_BLEND_ZERORenderTarget[0].BlendOpD3D11_BLEND_OP_ADDRenderTarget[0].SrcBlendAlphaD3D11_BLEND_ONERenderTarget[0].DestBlendAlphaD3D11_BLEND_ZERORenderTarget[0].BlendOpAlphaD3D11_BLEND_OP_ADDRenderTarget[0].RenderTargetWriteMaskD3D11_COLOR_WRITE_ENABLE_ALL Note that D3D11_BLEND_DESC is identical to {{D3D10_BLEND_DESC1}}.If the driver type is set to <see cref="SharpDX.Direct3D.DriverType.Hardware"/>, the feature level is set to less than or equal to <see cref="SharpDX.Direct3D.FeatureLevel.Level_9_3"/>, and the pixel formatofthe render target is set to <see cref="SharpDX.DXGI.Format.R8G8B8A8_UNorm_SRgb"/>, DXGI_FORMAT_B8G8R8A8_UNORM_SRGB, or DXGI_FORMAT_B8G8R8X8_UNORM_SRGB, the display device performs the blend in standard RGB (sRGB) space and not in linear space. However, if the feature level is set to greater thanD3D_FEATURE_LEVEL_9_3, the display device performs the blend in linear space.
/// </remarks>
/// <unmanaged>D3D11_BLEND_DESC</unmanaged>
public partial struct BlendDescription {
/// <summary>
/// Determines whether or not to use alpha-to-coverage as a multisampling technique when setting a pixel to a rendertarget.
/// </summary>
/// <unmanaged>BOOL AlphaToCoverageEnable</unmanaged>
public bool AlphaToCoverageEnable {
get {
return (_AlphaToCoverageEnable!=0)?true:false;
}
set {
_AlphaToCoverageEnable = value?1:0;
}
}
internal int _AlphaToCoverageEnable;
/// <summary>
/// Set to TRUE to enable independent blending in simultaneous render targets. If set to FALSE, only the RenderTarget[0] members are used. RenderTarget[1..7] are ignored.
/// </summary>
/// <unmanaged>BOOL IndependentBlendEnable</unmanaged>
public bool IndependentBlendEnable {
get {
return (_IndependentBlendEnable!=0)?true:false;
}
set {
_IndependentBlendEnable = value?1:0;
}
}
internal int _IndependentBlendEnable;
/// <summary>
/// An array of render-target-blend descriptions (see <see cref="SharpDX.Direct3D11.RenderTargetBlendDescription"/>); these correspond to the eight rendertargets that can be set to the output-merger stage at one time.
/// </summary>
/// <unmanaged>D3D11_RENDER_TARGET_BLEND_DESC RenderTarget[8]</unmanaged>
public SharpDX.Direct3D11.RenderTargetBlendDescription[] RenderTarget {
get {
if (_RenderTarget == null) {
_RenderTarget = new SharpDX.Direct3D11.RenderTargetBlendDescription[8];
}
return _RenderTarget;
}
}
internal SharpDX.Direct3D11.RenderTargetBlendDescription[] _RenderTarget;
// Internal native struct used for marshalling
[StructLayout(LayoutKind.Sequential, Pack = 0 )]
internal unsafe partial struct __Native {
public int _AlphaToCoverageEnable;
public int _IndependentBlendEnable;
public SharpDX.Direct3D11.RenderTargetBlendDescription RenderTarget;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget1;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget2;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget3;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget4;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget5;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget6;
SharpDX.Direct3D11.RenderTargetBlendDescription __RenderTarget7;
// Method to free native struct
internal unsafe void __MarshalFree()
{
}
}
// Method to marshal from native to managed struct
internal unsafe void __MarshalFrom(ref __Native @ref)
{
this._AlphaToCoverageEnable = @ref._AlphaToCoverageEnable;
this._IndependentBlendEnable = @ref._IndependentBlendEnable;
fixed (void* __to = &this.RenderTarget[0]) fixed (void* __from = &@ref.RenderTarget) SharpDX.Utilities.CopyMemory((IntPtr) __to, (IntPtr) __from, 8*sizeof ( SharpDX.Direct3D11.RenderTargetBlendDescription));
}
// Method to marshal from managed struct tot native
internal unsafe void __MarshalTo(ref __Native @ref)
{
@ref._AlphaToCoverageEnable = this._AlphaToCoverageEnable;
@ref._IndependentBlendEnable = this._IndependentBlendEnable;
fixed (void* __to = &@ref.RenderTarget) fixed (void* __from = &this.RenderTarget[0]) SharpDX.Utilities.CopyMemory((IntPtr) __to, (IntPtr) __from, 8*sizeof ( SharpDX.Direct3D11.RenderTargetBlendDescription));
}
}
</pre><br />
<ul><li><b>For Interfaces the mapping is quite complex</b>, because it is necessary to handle lost of different cases:</li>
<ul><li>Optionnal structure in input </li>
<li>Optionnal parameters</li>
<li>Output an array of interface</li>
<li>Perform some custom marshaling (for example, with the previous BlendDescription structure)</li>
<li> Generating properties for methods that are property elligible</li>
<li>...etc.</li>
</ul></ul>For example, the method using the BlendDescription is like this:<br />
<pre class="brush: csharp" name="code">/// <summary>
/// Create a blend-state object that encapsules blend state for the output-merger stage.
/// </summary>
/// <remarks>
/// An application can create up to 4096 unique blend-state objects. For each object created, the runtime checks to see if a previous object has the same state. If such a previous object exists, the runtime will return a pointer to previous instance instead of creating a duplicate object.
/// </remarks>
/// <param name="blendStateDescRef">Pointer to a blend-state description (see <see cref="SharpDX.Direct3D11.BlendDescription"/>).</param>
/// <param name="blendStateRef">Address of a pointer to the blend-state object created (see <see cref="SharpDX.Direct3D11.BlendState"/>).</param>
/// <returns>This method returns E_OUTOFMEMORY if there is insufficient memory to create the blend-state object. See {{Direct3D 11 Return Codes}} for other possible return values.</returns>
/// <unmanaged>HRESULT CreateBlendState([In] const D3D11_BLEND_DESC* pBlendStateDesc,[Out, Optional] ID3D11BlendState** ppBlendState)</unmanaged>
public SharpDX.Result CreateBlendState(ref SharpDX.Direct3D11.BlendDescription blendStateDescRef, out SharpDX.Direct3D11.BlendState blendStateRef){
unsafe {
SharpDX.Direct3D11.BlendDescription.__Native blendStateDescRef_ = new SharpDX.Direct3D11.BlendDescription.__Native();</pre><pre class="brush: csharp" name="code">blendStateDescRef.__MarshalTo(ref blendStateDescRef_);
IntPtr blendStateRef_ = IntPtr.Zero;
SharpDX.Result __result__;
__result__= (SharpDX.Result)SharpDX.Interop.CalliInt(_nativePointer, 20 * 4, &blendStateDescRef_, &blendStateRef_); </pre><pre class="brush: csharp" name="code"> blendStateDescRef.__MarshalFree(); </pre><pre class="brush: csharp" name="code">blendStateRef = (blendStateRef_ == IntPtr.Zero)?null:new SharpDX.Direct3D11.BlendState(blendStateRef_);
__result__.CheckError();
return __result__;
}
}
</pre><br />
In the previous example, you can see that the input BlendDescription structure is in fact marshalled to an intermediate native structure suitable for unmanaged code (internal __Native struct for BlendDescription). The marshall code is also responsible to free the native struct (if there are any allocations, like for strings).<br />
<br />
The marshalling has some nice optimizations, like for passing struct by value or by reference : All the methods in C++ are using a pointer for a struct (for getting and setting), but with the marshaller, we can decide if we want to have a struct passed by value or by ref. Currently, the generator is calculating the size of the valuetype. If the valuetype is less or equal 16 bytes, the valuetype is passed by value, otherwise it's passed by ref.<br />
<br />
A more standard interface with simple marshalling is like this: (Note for example the GUID integrated, the properties auto-generated from methods, and methods that are hidden from the public API)<br />
<br />
<pre class="brush: csharp" name="code">/// <summary>
/// This interface is used to return arbitrary length data.
/// </summary>
/// <unmanaged>ID3D10Blob</unmanaged>
[Guid("8ba5fb08-5195-40e2-ac58-0d989c3a0102")]
public partial class Blob : SharpDX.ComObject {
public Blob(IntPtr basePtr) : base(basePtr) {
}
/// <summary>
/// Get a pointer to the data.
/// </summary>
/// <unmanaged>void* GetBufferPointer()</unmanaged>
public IntPtr BufferPointer {
get { return GetBufferPointer(); }
}
/// <summary>
/// Get the size.
/// </summary>
/// <unmanaged>SIZE_T GetBufferSize()</unmanaged>
public SharpDX.Size BufferSize {
get { return GetBufferSize(); }
}
/// <summary>
/// Get a pointer to the data.
/// </summary>
/// <returns>Returns a pointer.</returns>
/// <unmanaged>void* GetBufferPointer()</unmanaged>
internal IntPtr GetBufferPointer() {
unsafe {
IntPtr __result__;
__result__= (IntPtr)SharpDX.Interop.CalliPtr(_nativePointer, 3 * 4);
return __result__;
}
}
/// <summary>
/// Get the size.
/// </summary>
/// <returns>The size of the data, in bytes.</returns>
/// <unmanaged>SIZE_T GetBufferSize()</unmanaged>
internal SharpDX.Size GetBufferSize() {
unsafe {
SharpDX.Size __result__;
__result__= (SharpDX.Size)SharpDX.Interop.CalliPtr(_nativePointer, 4 * 4);
return __result__;
}
}
}
</pre><br />
<br />
<ul><li><b>For functions, the mapping is quite straightforward</b>, because we are relying on a plain pinvoke interop. This was the easiest choice and easier to generate. Although pInvoke calls are still hidden in order to perform some parameter transformation, mostly in order to support the custom COM Object model generated. </li>
</ul><br />
A function call is generated like this:<br />
<pre class="brush: csharp" name="code">/// <unmanaged>HRESULT D3D11CreateDevice([In, Optional] IDXGIAdapter* pAdapter,[None] D3D_DRIVER_TYPE DriverType,[None] HMODULE Software,[None] D3D11_CREATE_DEVICE_FLAG Flags,[In, Buffer, Optional] const D3D_FEATURE_LEVEL* pFeatureLevels,[None] UINT FeatureLevels,[None] UINT SDKVersion,[Out,Optional] ID3D11Device** ppDevice,[Out, Optional] D3D_FEATURE_LEVEL* pFeatureLevel,[Out, Optional] ID3D11DeviceContext** ppImmediateContext)</unmanaged>
public static SharpDX.Result CreateDevice(SharpDX.DXGI.Adapter adapterRef, SharpDX.Direct3D.DriverType driverType, IntPtr software, SharpDX.Direct3D11.DeviceCreationFlags flags, SharpDX.Direct3D.FeatureLevel[] featureLevelsRef, int featureLevels, int sDKVersion, out SharpDX.Direct3D11.Device deviceRef, out SharpDX.Direct3D.FeatureLevel featureLevelRef, out SharpDX.Direct3D11.DeviceContext immediateContextRef) {
unsafe {
IntPtr deviceRef_ = IntPtr.Zero;
IntPtr immediateContextRef_ = IntPtr.Zero;
SharpDX.Result __result__;
__result__= (SharpDX.Result)D3D11CreateDevice_((adapterRef == null)?IntPtr.Zero:adapterRef.NativePointer, driverType, software, flags, featureLevelsRef, featureLevels, sDKVersion, out deviceRef_, out featureLevelRef, out immediateContextRef_);
deviceRef = (deviceRef_ == IntPtr.Zero)?null:new SharpDX.Direct3D11.Device(deviceRef_);
immediateContextRef = (immediateContextRef_ == IntPtr.Zero)?null:new SharpDX.Direct3D11.DeviceContext(immediateContextRef_);
__result__.CheckError();
return __result__;
}
}
/// <summary>Native Interop Function</summary>
/// <unmanaged>HRESULT D3D11CreateDevice([In, Optional] IDXGIAdapter* pAdapter,[None] D3D_DRIVER_TYPE DriverType,[None] HMODULE Software,[None] D3D11_CREATE_DEVICE_FLAG Flags,[In, Buffer, Optional] const D3D_FEATURE_LEVEL* pFeatureLevels,[None] UINT FeatureLevels,[None] UINT SDKVersion,[Out,Optional] ID3D11Device** ppDevice,[Out, Optional] D3D_FEATURE_LEVEL* pFeatureLevel,[Out, Optional] ID3D11DeviceContext** ppImmediateContext)</unmanaged>
[DllImport("d3d11.dll", EntryPoint = "D3D11CreateDevice", CallingConvention = CallingConvention.StdCall, PreserveSig = true), SuppressUnmanagedCodeSecurityAttribute]
private extern static SharpDX.Result D3D11CreateDevice_(IntPtr adapterRef, SharpDX.Direct3D.DriverType driverType, IntPtr software, SharpDX.Direct3D11.DeviceCreationFlags flags, SharpDX.Direct3D.FeatureLevel[] featureLevelsRef, int featureLevels, int sDKVersion, out IntPtr deviceRef, out SharpDX.Direct3D.FeatureLevel featureLevelRef, out IntPtr immediateContextRef);
</pre><h3>Extend the model in C#</h3><br />
All those classes are then integrated in a VS solution with 4 assemblies:<br />
<ul><li>A core assembly that contains non generated code (ComObject, DataStream, Vectors, Utilities...) and common enumeration and structs for Direct3D (structures that are usually shared between D3D10, D3D10.1 and D3D11).</li>
<li>An assembly for DXGI that has a dependency to the core assembly</li>
<li>An assembly for D3DCompiler that has a dependency to the core assembly</li>
<li>An assembly for D3D11 that has a dependency to the core, DXGI and D3DCompiler</li>
</ul>In order to quickly develop this new Wrapper, I have taken lots of portion of code from SlimDX, using the same design philosophy, mainly the <a href="http://code.google.com/p/slimmath/">Slim.Math </a>assembly in order to have all the Vectors and math functions ready-to-use. The only difference is that I have moved Vectors*/Matrix class to the main core, while still leaving higher level math classes to a separate Math assembly (BoudingSphere, Plain, intersection calculation... etc.)<br />
<br />
<br />
You may have noticed that all the generated class are tagged with the C# keyword "partial", making extension quite easy to integrate.<br />
<br />
Why do we need extensions? Well, Direct3D 11 API is sometimes not easy to use, there are a couple of redundancy that doesn't map well to C#. For example, methods are taking an array of structure + the size of this array => In C#, you would pass the array, and the size will be inferred from that... this is not strictly equivalent to C++, because you could pass an array larger than the number of elements you want to effectively pass, but this is the most common way the API is going to be used... so... <br />
<br />
For example, to create a DXGI Factory, you should have to call DXGICreateFactory... because we don't need to expose directly those functions, the DXGICreateFactory are tagged with internal keyword and I have added a new constructor to the DXGI Factory like this:<br />
<pre class="brush: csharp" name="code">using System;
using System.Runtime.InteropServices;
namespace SharpDX.DXGI
{
public partial class Factory
{
/// <summary>
/// Default Constructor for Factory
/// </summary>
public Factory() : base(IntPtr.Zero)
{
IntPtr factoryPtr;
DXGI.CreateDXGIFactory(GetType().GUID, out factoryPtr);
NativePointer = factoryPtr;
}
</pre><br />
Finally in a assembly project, you have:<br />
<ul><li>Generated classes : Enumerations.cs, Structures.cs, Interfaces.cs, Functions.cs</li>
<li> Extension classes : They are placed in a subdirectory Extension with the filename of the extended class .e.g. Factory.cs</li>
<li>Non generated classes : For example, VertexBufferBinding which is used by a custom SetVertexBuffers in order to set strides, offsets and buffers in a more friendly way like :</li>
</ul><pre class="brush: csharp" name="code">context.InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(vertices, 32, 0));
</pre><h3>Example of ported SlimDX MiniTri sample</h3><br />
Here is a port of <a href="http://code.google.com/p/slimdx/source/browse/trunk/samples/Direct3D11/MiniTri11/Program.cs">MiniTri D3D11 sample </a>to this new API. You could verify that the API is really close to SlimDX experience...<br />
<pre class="brush: csharp" name="code">using System;
using SharpDX;
using SharpDX.Direct3D;
using SharpDX.Direct3D11;
using SharpDX.DXGI;
using SharpDX.Windows;
using SharpDX.D3DCompiler;
using Buffer = SharpDX.Direct3D11.Buffer;
using Device = SharpDX.Direct3D11.Device;
namespace MiniTri
{
/// <summary>
/// SharpDX port of SlimDX-MiniTri Direct3D 11 Sample
/// </summary>
static class Program
{
[STAThread]
static void Main()
{
var form = new RenderForm("SharpDX - MiniTri Direct3D 11 Sample");
// SwapChain description
var desc = new SwapChainDescription()
{
BufferCount = 1,
BufferDescription = new ModeDescription(form.ClientSize.Width, form.ClientSize.Height, new Rational(60, 1), Format.R8G8B8A8_UNorm),
Windowed = true,
OutputWindow = form.Handle,
SampleDescription = new SampleDescription(1,0),
SwapEffect = SwapEffect.Discard,
BufferUsage = Usage.RenderTargetOutput
};
// Create Device and SwapChain
Device device;
SwapChain swapChain;
Device.CreateWithSwapChain(DriverType.Hardware, DeviceCreationFlags.Debug, desc, out device, out swapChain);
var context = device.ImmediateContext;
// Ignore all windows events
Factory factory = swapChain.GetParent<Factory>();
factory.MakeWindowAssociation(form.Handle, WindowAssociationFlags.IgnoreAll);
// New RenderTargetView from the backbuffer
Texture2D backBuffer = Texture2D.FromSwapChain<Texture2D>(swapChain, 0);
var renderView = new RenderTargetView(device, backBuffer);
// Compile Vertex and Pixel shaders
var vertexShaderByteCode = ShaderBytecode.CompileFromFile("MiniTri.fx", "VS", "vs_4_0", ShaderFlags.None, EffectFlags.None);
var vertexShader = new VertexShader(device, vertexShaderByteCode);
var pixelShaderByteCode = ShaderBytecode.CompileFromFile("MiniTri.fx", "PS", "ps_4_0", ShaderFlags.None, EffectFlags.None);
var pixelShader = new PixelShader(device, pixelShaderByteCode);
// Layout from VertexShader input signature
var layout = new InputLayout(device, ShaderSignature.GetInputSignature(vertexShaderByteCode), new[] {
new InputElement("POSITION", 0, Format.R32G32B32A32_Float, 0, 0),
new InputElement("COLOR", 0, Format.R32G32B32A32_Float, 16, 0)
});
// Write vertex data to a datastream
var stream = new DataStream(32 * 3, true, true);
stream.WriteRange(new[] {
new Vector4(0.0f, 0.5f, 0.5f, 1.0f), new Vector4(1.0f, 0.0f, 0.0f, 1.0f),
new Vector4(0.5f, -0.5f, 0.5f, 1.0f), new Vector4(0.0f, 1.0f, 0.0f, 1.0f),
new Vector4(-0.5f, -0.5f, 0.5f, 1.0f), new Vector4(0.0f, 0.0f, 1.0f, 1.0f)
});
stream.Position = 0;
// Instantiate Vertex buiffer from vertex data
var vertices = new Buffer(device, stream, new BufferDescription()
{
BindFlags = BindFlags.VertexBuffer,
CPUAccessFlags = CpuAccessFlags.None,
MiscFlags = ResourceOptionFlags.None,
SizeInBytes = 32 * 3,
Usage = ResourceUsage.Default,
StructureByteStride = 0
});
stream.Release();
// Prepare All the stages
context.InputAssembler.InputLayout = layout;
context.InputAssembler.PrimitiveTopology = PrimitiveTopology.Trianglelist;
context.InputAssembler.SetVertexBuffers(0, new VertexBufferBinding(vertices, 32, 0));
context.VertexShader.Set(vertexShader);
context.Rasterizer.SetViewports(new Viewport(0, 0, form.ClientSize.Width, form.ClientSize.Height, 0.0f, 1.0f));
context.PixelShader.Set(pixelShader);
context.OutputMerger.SetTargets(renderView);
// Main loop
MessagePump.Run(form, () =>
{
context.ClearRenderTargetView(renderView, new Color4(1.0f, 0.0f, 0.0f, 0.0f));
context.Draw(3, 0);
swapChain.Present(0, PresentFlags.None);
});
// Release all resources
vertexShaderByteCode.Release();
vertexShader.Release();
pixelShaderByteCode.Release();
pixelShader.Release();
vertices.Release();
layout.Release();
renderView.Release();
backBuffer.Release();
context.ClearState();
context.Flush();
device.Release();
context.Release();
swapChain.Release();
factory.Release();
}
}
}
</pre><h2>Next?</h2><br />
Wow, this was not supposed to be a so long post! I have been a bit into the internals of the generator and It may not be interesting for a general audience, but at least I have taken some time to put this down on a paper, to clarify things.<br />
<br />
Although, I have not detailled everything. For example, you have probably noticed from the previous example that I'm still not using the D3D11 Effects11 API. Well, the problem is that Microsoft has removed the Effects API from D3D11. Why? Probably because the code is hidding too much about how you could interact properly (and more effitiently) with D3D11 API. But this is one decision I don't fully agree : Look at XNA 4.0 : They have removed the used of VertexShader, PixelShader directly in favor of Effects classes... In one API, they are no longer supporting it, in another, they are making it the only and mandatory one... Some could argue that XNA doesn't have the same target... but still, from a software design perspective, I'm quite <span class="short_text" id="result_box" lang="en"><span style="color: black;" title="">doubtful.</span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title=""> </span></span><span class="short_text" id="result_box" lang="en"><span style="color: black;" title=""> </span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title="">The great news is that looking at the C++ Effects11 sample, I have been able to port the most interesting part : decoding an Effect bytecode to extract usefull information, like constant buffers, techniques, stages, shader's bytecodes...etc. I'm not going to support the whole fx_5_0 profile, because I'm usually using a subset of this : for example, I don't find practical to declare samplers state, blending...etc. in the shader and I do prefer to have them instantiated from the C# code. On the other hand, I like a lot the way the Effects library is encapsulating constant buffer and shader resource view binding to shader stages. This is one of the most laborious things to do if you are going with the raw Direct3D 11 interface. So if I could have an Effect framework supporting at least techniques, pass and proper automatic constant buffer and SRV bindings, I would be very happy. This part will deserve another post!</span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title=""> </span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title="">Also, working more with SlimDX and this new API wrapper, I have been working with a XNA like API on top of a Direct3D 11 API, and It was in fact really easy to achieve (of course, without the content pipeline, which is the true benefit of XNA). Why do we need such a higher API? Well, Direct3D 11 is really powerful with its buffer/resource management, but the fact is that it's <i>much </i>more verbose. But think about it : When you use a Texture2D, you will need most of the time a ShaderResourceView on it.... If you want a texture2D as a render target, you will probably need a RenderTargetView, and because It's a RenderTarget, you will probably use this RenderTarget as a ShaderResourceView for another pass... So in the end, there are lots of things that can be handled in the background, even if you are using a Direct3D 11 API. The nice thing about this kind of API is that you can play with some geometry or compute shaders, while still having the pleasure to work with a high level API. This will also probably be part of a post!</span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title=""><br />
</span></span><br />
<span class="short_text" id="result_box" lang="en"><span style="color: black;" title="">So, what's next? I just finished the mapping and the port of the MiniTri yesterday. The current wrapper is probably not yet fully usable and doesn't have the same level of API richness than SlimDX. Threre are still lots of -small- extensions code to add to make the coding experience better than a somewhat raw D3D11 API. Within the next days, I'm going to play much more with this new wrapper and see how far can it go...</span></span><br />
<br />
<span style="font-size: x-small;"><i>(note: 1st draft version of this document)</i></span>xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com4tag:blogger.com,1999:blog-1076643699683521890.post-90335603651005474232010-08-26T00:28:00.021+11:002011-11-25T00:09:37.852+11:00Making of Ergon 4K PC Intro<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhGLMHKFu8KKIfYSywqPaJ-21iQVqWWvmoSKk5CmzLNPl6oxyhqPcX5oVbkY4dzG_-JEMSNRmvZk9ur42xrzxYrgJvboNCw5_9RstDcpWD5uQF3D2KhW0DWeMD0HBcM2vxMEHBKBvmqT8/s1600/ergon+2010-03-11+23-34-21-21.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhhGLMHKFu8KKIfYSywqPaJ-21iQVqWWvmoSKk5CmzLNPl6oxyhqPcX5oVbkY4dzG_-JEMSNRmvZk9ur42xrzxYrgJvboNCw5_9RstDcpWD5uQF3D2KhW0DWeMD0HBcM2vxMEHBKBvmqT8/s320/ergon+2010-03-11+23-34-21-21.jpg" /></a></div>
You are not going to discover any fantastic trick here, the intro itself is not an outstanding coding performance, but I always enjoy reading the making of other intros, so It's time to take some time to put this on a paper!<br />
<br />
What is <a href="http://pouet.net/prod.php?which=54529">Ergon</a>? It's a small 4k intro (meaning 4096 byte executable) that was released at the <a href="http://breakpoint.untergrund.net/">2010 Breakpoint demoparty</a> (if you can't run it on your hardware, you can still <a href="http://www.youtube.com/watch?v=i_feg-bpqTY">watch it on youtube</a>), which surprisingly was able to finish to the 3rd place! I did the coding, design and worked also on the music with my friend ulrick.<br />
<br />
That was a great experience even if I didn't expect to work on this production at the beginning of the year... but at the end of January, when BP2010 was announced and supposed to be the last one, I was motivated to go there, and why not, release a 4k intro! One month and a half later, the demo was almost ready... wow, 3 weeks before the party, first time to finish something so ahead an event! But yep, I was able to work on it on part time during the week (and the night of course)... But when I started on it, I had no idea where this project would bring me to... or even what kind of 3D API I had to start from doing this intro!<br />
<br />
<a name='more'></a><h3>
OpenGL, DirectX 9, 10 or 11?</h3>
<br />
At FRequency, xt95 is mainly working in OpenGL, mostly due to the fact that he is a linux user. All our previous intros were done using OpenGL, although I did provide some help on some intros, bought OpenGL books few years ago... I'm not a huge fan of the OpenGL C API, but most importantly, from my short experience on this, I was always able to better strip down DirectX code size than OpenGL code... At that time, I was also working a bit more on DirectX API... I even bought a 5770 ATI earlier to be able to play with D3D11 Compute Shader api... I'm also mostly a windows user... DirectX has a very well integrated documentation in Visual Studio, a good SDK, lots of samples inside, a cleaner API (more true on recent D3D10/D3D11), some cool tools like PIX to debug shaders... and thought also that programming on DirectX on windows might reduce the risk to get some incompatibilities between NVidia and ATI graphics card (although, I found that, at least with D3D9, this is not always true...).<br />
<br />
So ok, DirectX was selected... but which version? I started my first implementation with D3D10. I know that the code is much more verbose than D3D9 and OpenGL2.0, but I wanted to practice it a bit more the somehow "new" API than just reading a book about it. I was also interested to plug some text in the demo and tried an integration with latest Direct2D/DirectWrite API.<br />
<br />
Everything went well at the beginning with D3D10 API. The code was clean, thanks to the thin layer I developed around DirectX to make the coding experience much closer to what I use to have in C# with SlimDx for example. The resulting C++ code was something like this :<br />
<pre class="brush: csharp" name="code">//
// Set VertexBuffer for InputAssembler Stage
device.InputAssembler.SetVertexBuffers(screen.vertexBuffer, sizeof(VertexDataOffline));
// Set TriangleList PrimitiveTopology for InputAssembler Stage
device.InputAssembler.SetPrimitiveTopology(PrimitiveTopology::TriangleStrip);
// Set VertexShader for the current Pass
device.VertexShader.Set(effect.vertexShader);
</pre>
Very pleasant to develop with it, but because I wanted to test D2D1, I switched to D3D10.1 which can be configured to run on D3D10 hardware (with the feature level thing)... So I also started to slightly wrap up the Direct2D API and was able to produce very easily some really nice text... but wow... the code was a bit too large for a 4k (but would be perfect for a 64k). <br />
<br />
Then during this experiment phase, I tried the D3D11 API with the Compute Shader thing... and found that the code is much more compact than D3D10 if you are performing some kind of... for example, raymarching... I didn't compare code size, but I suspect the code to be able to compete with its D3D9 counterpart (although, there is a downside in D3D11 : if you can afford a real D3D11 hardware, a compute shader can directly render to the screen buffer... otherwise, using the D3D11 Compute shader with features level 10, you have to copy the result from one resource to another... which might hit the size benefit...).<br />
<br />
I was happy to see that the switch to D3D11 was easy, with some continuity from D3D10 on the API "look & feel"... Although I was disappointed to learn that working this D3D11 and D2D1 was not straightforward because D2D1 is only compatible with D3D10.1 API (which you can run with feature level 9.0 to 10), forcing to initialize and maintain two devices (one for D3D10.1 and one for D3D11), playing with DXGI shared resource between the devices... wow, lots of work, lots of code... and of course, out of question for a 4k...<br />
<br />
So I tried... a plain old good D3D9... and that was of course much compact in size than their D3D10 counterpart... So for around two weeks in February, I played with those various API while implementing some basic scene for the intro.I just had a bad surprise when releasing the intro, because lots of people were not able to run it : weird because I was able to test it on several NVidias and at least my ATI 5770... I didn't expect D3D9 to be so sensitive to that, or at least, a bit less sensitive than OpenGL... but I was wrong.<br />
<br />
<h3>
Raymarching optimization</h3>
<br />
I decided to go for an intro using the <a href="http://code4k.blogspot.com/2009/10/potatro-and-raymarching-story-of.html">raymarching algorithm</a> that was more likely to be able to deliver a "fat" content in a tiny amount of code. Although, the raymarching stuff was already a bit in the "retired", after the fantastic intros released earlier in 2009 (<a href="http://pouet.net/prod.php?which=52938">Elevated</a> - not really a raymarching intro but soo impressive!, <a href="http://www.pouet.net/prod.php?which=52940">Sult</a>, <a href="http://pouet.net/prod.php?which=53937">Rudebox</a>, <a href="http://www.pouet.net/prod.php?which=53605">Muon-Baryon</a>...etc). But I didn't have enough time to explore a new effect and was not even confident to be able to find anything interesting at that time... so... ok, raymarching.<br />
<br />
So for one week, after building a 1st scene, I spent my time to try to optimize the raymarching algo. There was an instructive thread on pouet about this : "<a href="http://www.pouet.net/topic.php?which=6675&page=1&x=21&y=7">So, what do distance field equations look like? And how do we solve them?</a>". I tried to implement some trick like...<br />
<ol>
<li>Generate grid on the vertex shader (with 4x4 pixels for example), to precompute a raw view of the scene, storing the minimal distance step to go before hitting a surface... let the pixel shader to get those interpolate distances (multiplied by a small reduction factor like .9f) and perform some fine grained raymarching with fewer iterations</li>
<li>Generate a pre-rendered 3D volume of the scene at a much lower density (like 96x96x96) and use this map to navigate in the distance fields while still performing some "sphere tracing" refinement if needed</li>
<li>I tried also somekind of level of detail on the scene : for example, instead of having a texture lookup (for the "bump mapping") for each step during the raymarching, allow the raymarcher to use a simplified analytical surface scene and switch to the more detailled one for the last step</li>
</ol>
Well, I have to admit that all those techniques were not really clever in anyway... and the result was matching the lack of this cleverness! None of them provide a significant speed optimization compare to the code size hit they were generated.<br />
<br />
So after one week of optimization, well, I just went to a basic raymarcher algo. The shader was developed under Visual C++, integrated in the project (thanks to NShader syntax highlighting). I did a small C# tool to strip the shader comments, remove unnecessary spaces... integrated in the build (pre-build events in VC++), It's really enjoyable to work with this toolchain.<br />
<br />
<h3>
Scenes design</h3>
<br />
For the scenes, I decided to use the same kind of technique used in the Rudebox 4k intro : Leveraging more on the geometry and lights, but not on the materials. That made the success of the rudebox and I was motivated to build some complex CSG with boolean operations on basic elements (box, sphere...etc.). The nice thing about this approach is that It avoids to use inside the ISO surface anykind of if/then/else for determining the material... just letting the lights properly set in the scene might do the work. Yep, indeed, rudebox is for example a scene with say, a white material for all the objects. What makes the difference is the position of lights in the scene, their intensity...etc. Ergon used the same trick here.<br />
<br />
I spent around two to three weeks to build the scenes. I ended up with 4 scenes, each one quite cool on their own, with a consistent design among them. One of the scene was using the fonts to render a wall of text in raymarching.<br />
<br />
Because I'm not sure that I will be able to use those scenes, well, I'm going to post their screenshot here!<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF-fxWTZ1PEUGrkt5w_lVy1QgqNQqdqiAMaNzbkccQkW9irOaPzj1Sv0wHAC1wpn0kPujtjT7gX9mqu_B9MIiOUPIRj6odHyCKkk3m_xjLQ8rP4EavEydwzaaPCemAuVaVGe4nztY7yHY/s1600/centroid+2010-02-15+22-50-35-95.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhF-fxWTZ1PEUGrkt5w_lVy1QgqNQqdqiAMaNzbkccQkW9irOaPzj1Sv0wHAC1wpn0kPujtjT7gX9mqu_B9MIiOUPIRj6odHyCKkk3m_xjLQ8rP4EavEydwzaaPCemAuVaVGe4nztY7yHY/s320/centroid+2010-02-15+22-50-35-95.jpg" /></a></div>
<br />
The <b>1st scene</b> I developed during my D3D9/D3D10/D3D11 API experiments was a massive tentacle model coming from a balckhole. All the tentacles were moving around a weird cutted sphere, with a central "eye"... I was quite happy about this scene that had a unique design. From the beginning, I wanted to add some post-processing, to enhance the visuals, and to make them a bit different from other raymarching scene... So I went with a simple post-processing that was performing some patterns on the pixels, adding a radial blur to produce some kind of "ghost rays" coming out from the scene, making the corners darker, and adding a small flickering the more you go to the corners. Well, only this piece of code was already taking a scene on its own, but that was the price to have a genuine ambiance, so...<br />
<br />
The colors and theming was almost settled from the beginning... I'm a huge fan of warm colors!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiRTRcwt-phxd9XUE1k8hriDoqkEYlWk4SfMOBZ0SQKD4-ygEPZno0KaU-lvn5YEPg9VSsBXyN_FYF3nInuYD_A0uWovECTHURFABC1VTWZBb8J5H6MIglz6FRXkFSX231ECoZWSsoJTk/s1600/centroid_logo_FRequency+2010-02-15+22-50-55-35.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiRTRcwt-phxd9XUE1k8hriDoqkEYlWk4SfMOBZ0SQKD4-ygEPZno0KaU-lvn5YEPg9VSsBXyN_FYF3nInuYD_A0uWovECTHURFABC1VTWZBb8J5H6MIglz6FRXkFSX231ECoZWSsoJTk/s320/centroid_logo_FRequency+2010-02-15+22-50-55-35.jpg" /></a></div>
The <b>2nd scene</b> was using a font rendering coupling with the raymarcher.... a kind of flying flag, with the logo FRequency appearing from left to right with a light on it... (I will probably release those effects on pouet just for the record...), that was also a fresh use of raymarching... didn't see anything like this in latest 4k production, so, I was expecting to insert this text in the 4k, as It's not so common... The code to use the d3d font was not too fat... so I was still confident to be able to use those 2 scenes.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7wWo5DrCUAcBUVr-6KNTNpnfC3w2vAEey8kQvOD1pUeZ3Mb2ncsgZ8EgY25WMH-jntfiq3GlpvZrBd4MwixPbnoPTEWVJnzn3ZfaeJU7i7ndUHP2GVucTIrk7uLGLaSqQa_l8aKoXDio/s1600/centroid_raptor+2010-08-25+14-01-40-63.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7wWo5DrCUAcBUVr-6KNTNpnfC3w2vAEey8kQvOD1pUeZ3Mb2ncsgZ8EgY25WMH-jntfiq3GlpvZrBd4MwixPbnoPTEWVJnzn3ZfaeJU7i7ndUHP2GVucTIrk7uLGLaSqQa_l8aKoXDio/s320/centroid_raptor+2010-08-25+14-01-40-63.jpg" /></a></div>
After that, I was looking for some nasty objects... so for the <b>3rd scene</b>, I tried to randomly play with some weird functions and ended up with a kind of "raptor" creature... I wanted also to use a weird generated texture I found few month ago, that was perfect for it. <br />
<br />
Finally, I wanted to use the texture to make a kind of lava sea with a moving snake on it... that was the last scene I coded (and of course, 2 others scenes that are too ugly to show here! :) ).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbY9Hxvo-wdo3lVuhDFgGKdtIMivy5bFk5IksUHCgtlz4pd2MRxS6LeDd1yWsaPb4YbkDfCoznC0psAV2it0mxDagCCx45p_Gv79SYYX3k0xxjUrh7Qua3lhNW8WZGdDoUK9vkTzxPFAo/s1600/ergon+2010-03-11+23-33-44-65.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbY9Hxvo-wdo3lVuhDFgGKdtIMivy5bFk5IksUHCgtlz4pd2MRxS6LeDd1yWsaPb4YbkDfCoznC0psAV2it0mxDagCCx45p_Gv79SYYX3k0xxjUrh7Qua3lhNW8WZGdDoUK9vkTzxPFAo/s320/ergon+2010-03-11+23-33-44-65.jpg" /></a></div>
<br />
We also started at that time, in February, to work on the music, and as I explained in my <a href="http://code4k.blogspot.com/2010/02/coding-4k-intro-for-breakpoint-2010.html">earlier posts</a>, we used 4klang synth for the intro. But making all those scenes with a music prototype, the "crinklered" compressed exe was more around 5ko... even If the shader code was already optimized in size, using some kind of preprocessor templating (like in rudebox or receptor). The intro was of course laking a clear direction, there was no transitions between the scenes... and most importantly, It was not possible to fit all those scenes in 4k, while expecting the music to grow a little bit more in the final exe...<br />
<br />
<h3 id="wormlava">
The story of the Worm-Lava texture </h3>
<br />
Last year, around November, while I was playing with several perlin's like noise, I found an interesting variation using perlin noise and the <a href="http://www.noisemachine.com/talk1/23.html">marble-cosine effect</a> that was able to represents some kind of worms, quite freaking ugly in some way, but that was a unique texture effect!<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMwBezBELGR52lneQw47YaEWXsGEwWp0wq6o6tdbkYCFGys4CnSMSpeeHG9_IXHAiM2wGrjM2DOIVAXZWmbugKpC7M8nwpr84R4ABb9RC3eC1YGmYvrKqX-Z2NoI8HVY2hyqm_bNG4sdM/s1600/BodyNoise.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMwBezBELGR52lneQw47YaEWXsGEwWp0wq6o6tdbkYCFGys4CnSMSpeeHG9_IXHAiM2wGrjM2DOIVAXZWmbugKpC7M8nwpr84R4ABb9RC3eC1YGmYvrKqX-Z2NoI8HVY2hyqm_bNG4sdM/s640/BodyNoise.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(Click to enlarge, lots of details in it!)</td></tr>
</tbody></table>
<br />
This texture was primarily developed in C# but the code was quite straightforward to port in a texture shader... Yep, that's probably an old trick with D3D9 to use the function <a href="http://msdn.microsoft.com/en-us/library/bb172834%28v=VS.85%29.aspx">D3DXFillTextureTX </a>to directly fill a texture from a shader with a single line of code... Why using this? Because It was the only way to get a noise() function accessible from a shader, without having to implement it... As weird as it may sounds, the <a href="http://msdn.microsoft.com/en-us/library/bb509629%28VS.85%29.aspx">HLSL perlin noise()</a> function is not accessible outside a texture shader. A huge drawback of this method is also that the shader is not a real GPU shader, but is instead computed on the CPU... that explain why ergon intro is taking so long to generate the texture at the beginning (with a 1280x720 texture resolution for example).<br />
<br />
So how does look this texture shader in order to generate this texture?<br />
<pre class="brush: csharp" name="code">// -------------------------------------------------------------------------
// worm noise function
// -------------------------------------------------------------------------
#define ty(x,y) (pow(.5+sin((x)*y*6.2831)/2,2)-.5)
#define t2(x,y) ty(y+2*ty(x+2*noise(float3(cos((x)/3)+x,y,(x)*.1)),.3),.7)
#define tx(x,y,a,d) ((t2(x, y) * (a - x) * (d - y) + t2(x - a, y) * x * (d - y) + t2(x, y - d) * (a - x) * y + t2(x - a, y - d) * x * y) / (a * d))
float4 x( float2 x : position, float2 y : psize) : color {
float a=0,d=64;
// Modified FBM functions to generate a blob texture
for(;d>=2;d/=2)
a += abs(tx(x.x*d,x.y*d,d,d)/d);
return a*2;
}
</pre>
<br />
The tx macro is basically applying a tiling on the noise. <br />
The core t2 and ty macros are the one that are able to generate this "worm-noise". It's in fact a tricky combination of the usual cosine perlin noise. Instead of having something like cos(x + noise(x,y)), I have something like special_sin( y + special_sin( x + noise(cos(x/3)+x,y), power1), power2), with special_sin function like ((1 + sin(x*power*2*PI))/2) ^ 2<br />
<br />
Also, don't be afraid... this formula didn't came out of my head like this... that was clearly after lots of permutations from the original function, with lots of run/stop/change_parameters steps! :D<br />
<br />
<h3>
Music and synchronization</h3>
<br />
It took some time to build the music theme and to be satisfied with it... At the beginning, I let ulrick making a first version of the music... But because I had a clear view of the design and direction, I was expecting a very specific progression in the tune and even in the chords used... That was really annoying for ulrick (excuse-me my friend!), as I was very intrusive in the composition process... At some point, I ended up in making a 2 pattern example of what I wanted in terms of chords and musical ambiance... and ulrick was kind enough to take this sample pattern and clever to add some intro's musical feeling in it. He will be able to talk about this better than me, so I'll ask him if he can insert some small explanation here!<br />
<br />
ulrick here:<i> « working with @lx on this prod was a very enjoyable job. I started a music which @lx did not like very much, it did not reflect the feelings that @lx wanted to give through the Ergon. He thus composed a few patterns using a very emotional musical scale. I entered into the music very easily and added my own stuffs. For the anecdote, I added a second scale to the music to allow for a clearer transition between the first and second parts of the Ergon. After doing so, we realized that our music actually used the chromatic scale on E</i> »<br />
<br />
The synchronization was the last part of the work in the demo. I first used the default synchronization mechanism from the 4klang... but I was lacking some features like, if the demo is running slowly, I needed to know exactly where I was... Using plain 4klang sync, I was missing some events on slow hardware, even preventing the intro to switch between the scenes, because the switching event was missed by the rendering loop!<br />
<br />
So I did my own small synchronization based on regular events of the snare and a reduce view of the sample patterns for this particular events. This is the only part of the intro that was developed in x86 assembler in order to keep it as small as possible.<br />
<br />
The whole code was something like this :<br />
<pre class="brush: cpp" name="code">static float const_time = 0.001f;
static int SAMPLES_PER_DRUMS = SAMPLES_PER_TICK*16;
static int SAMPLES_PER_DROP_DRUMS = SAMPLES_PER_TICK*4;
static int SMOOTHSTEP_FACTOR = 3;
static unsigned char drum_flags[96] = {
// pattern n° time z.z sequence
1,1,1,1, // pattern 0 0 0 0
1,1,1,1, // pattern 1 7,384615385 4 1
0,0,0,0, // pattern 2 14,76923077 8 2
0,0,0,0, // pattern 3 22,15384615 12 3
0,0,0,0, // pattern 4 29,53846154 16 4
0,0,0,0, // pattern 5 36,92307692 20 5
0,0,0,0, // pattern 6 44,30769231 24 6
0,0,0,0, // pattern 7 51,69230769 28 7
0,0,0,1, // pattern 8 59,07692308 32 8
0,0,0,1, // pattern 8 66,46153846 36 9
1,1,1,1, // pattern 9 73,84615385 40 10
1,1,1,1, // pattern 9 81,23076923 44 11
1,1,1,1, // pattern 10 88,61538462 48 12
0,0,0,0, // pattern 11 96 52 13
0,0,0,0, // pattern 2 103,3846154 56 14
0,0,0,0, // pattern 3 110,7692308 60 15
0,0,0,0, // pattern 4 118,1538462 64 16
0,0,0,0, // pattern 5 125,5384615 68 17
0,0,0,0, // pattern 6 132,9230769 72 18
0,0,0,0, // pattern 7 140,3076923 76 19
0,0,0,1, // pattern 8 147,6923077 80 20
1,1,1,1, // pattern 12 155,0769231 84 21
1,1,1,1, // pattern 13 162,4615385 88 22
};
// Calculate time, synchro step and boom shader variables
__asm {
fild dword ptr [time] // st0 : time
fmul dword ptr [const_time] // st0 = st0 * 0.001f
fstp dword ptr [shaderVar.x] // shaderVar.x = time * 0.001f
mov eax, dword ptr [MMTime.u.sample]
cdq
sub eax, SAMPLES_PER_TICK*8
jae not_first_drum
xor eax,eax
not_first_drum:
idiv dword ptr [SAMPLES_PER_DRUMS] // eax = drumStep , edx = remainder step
mov dword ptr [drum_step], eax
fild dword ptr [drum_step]
fstp dword ptr [shaderVar.z] // shaderVar.z = drumStep
not_end: cmp byte ptr [eax + drum_flags],0
jne no_boom
mov eax, SAMPLES_PER_TICK*4
sub eax,edx
jae boom_ok
xor eax,eax
boom_ok:
mov dword ptr [shaderVar.y],eax
fild dword ptr [shaderVar.y]
fidiv dword ptr [SAMPLES_PER_DROP_DRUMS] // st0 : boom
fild dword ptr [SMOOTHSTEP_FACTOR] // st0: 3, st1-4 = boom
fsub st(0),st(1) // st0 : 3 - boom , st1-3 = boom
fsub st(0),st(1) // st0 : 3 - boom*2, st1-2 = boom
fmul st(0),st(1) // st0 : boom * (3-boom*2), st1 = boom
fmulp st(1),st(0)
fstp dword ptr [shaderVar.y]
no_boom:
};
</pre>
<br />
That was smaller then what I was able to do with pure 4klang sync... with the drawback that the sync was probably too simplistic... but I couldn't afford more code for the sync... so...<br />
<br />
<h3>
Final mixing</h3>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrg3UVX6DcjUU5RBdqVJLVh0EAJcCfLpZ99M2hPdFcjKRPbK4h93ggAxIkM4S-1JzgNaMBmniZYe90nxrNBQVijN3DJ4LL9ukjysUFIsRAyQEq26idRg2eW3q4LP0PlvVkT6qhocax_nA/s1600/ergon+2010-03-11+23-34-41-70.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrg3UVX6DcjUU5RBdqVJLVh0EAJcCfLpZ99M2hPdFcjKRPbK4h93ggAxIkM4S-1JzgNaMBmniZYe90nxrNBQVijN3DJ4LL9ukjysUFIsRAyQEq26idRg2eW3q4LP0PlvVkT6qhocax_nA/s320/ergon+2010-03-11+23-34-41-70.jpg" /></a></div>
Once the music was almost finished, I spent a couple of days to work on the transitions, sync, camera movements. Because It was not possible to fit the 4 scenes, I had to mix the scene 3 (the raptor) and 4 (the snake and the lava sea), found a way to put a transition through a "central brain". Ulrick wanted to put a different music style for the transition, I was not confident with it... until I put the transition in action, letting the brain collapsed while the space under it was digging all around... and the music was fitting very well! cool!<br />
<br />
<br />
I did also use a simple big shader for the whole intro, with some if (time < x) then scene_1 else scene_2...etc. I didn't expect to do this, because this is hurting the performance in the pixel shader to do this kind of branch processing... But I was really running out of space here and the only solution was in fact to use a single shader with some repetitive code. Here is an excerpt from the shader code : You can see how scene and camera management has been done, as well as for lights. This part was compressing quite well due to its repetitive pattern. <br />
<pre class="brush: hlsl" name="code">// -------------------------------------------------------------------------
// t3
// Helper function to rotate a vector. Usage :
// t3(mypoint.xz, .7); <= rotate mypoint around Y axis with .7 radians
// -------------------------------------------------------------------------
float2 t3(inout float2 x,float y){
return x=x*cos(y)+sin(y)*float2(-x.y,x.x);
}
// -------------------------------------------------------------------------
// v : main raymarching function
// -------------------------------------------------------------------------
float4 v(float2 x:texcoord):color{
float a=1,b=0,c=0,d=0,e=0,f=0,i;
float3 n,o,p,q,r,s,t=0,y;
int w;
r=normalize(float3(x.x*1.25,-x.y,1)); // ray
x = float2(.001,0); // epsilon factor
// Scene management
if (z.z<39) {
w = (z.z<10)?0:(z.z>26)?3+int(fmod(z.z,5)):int(fmod(z.z,3));
//w=4;
if (w==0) { p=float3(12,5+30*smoothstep(16,0,z.x),0);t3(r.yz,1.1*smoothstep(16,0,z.x));t3(r.xz,1.54); }
if (w==1) { p=float3(-13,4,-8);t3(r.yz,.2);t3(r.xz,-.5);t3(r.xy,sin(z.x/3)/3); }
if (w==2) { p=float3(0,8.5,-5);t3(r.yz,.2);t3(r.xy,sin(z.x/3)/5); }
if (w==3) {
p=float3(13+sin(z.x/5)*3,10+3*sin(z.x/2),0);
t3(r.yz, sin(z.x/5)*.6);
t3(r.xz, 1.54+z.x/5);
t3(r.xy, cos(z.x/10)/3);
t3(p.xz,z.x/5);
}
if (w == 4) {
p=float3(30+sin(z.x/5)*3,8,0);
t3(r.yz, sin(z.x/5)/5);
t3(r.xz, 1.54+z.x/3);
t3(r.xy, sin(z.x/10)/3);
t3(p.xz,z.x/3);
}
if (w > 4) {
p=float3(4.5,25+10*sin(z.x/3),0);
t3(r.yz, 1.54*sin(z.x/5));
t3(r.xz, .7+z.x/2);
t3(r.xy, sin(z.x/10)/3);
t3(p.xz,z.x/2);
}
} else if (z.z<52) {
p=float3(20,20,0);
t3(r.yz, .9);
t3(r.xz, 1.54+z.x/4);
t3(p.xz,z.x/4);
} else if (z.z<81) {
w = int(fmod(z.z,3));
if (w==0 ) {
p=float3(40+sin(z.x/5)*3,8,0);
t3(r.yz, sin(z.x/5)/5);
t3(r.xz, 1.54+z.x/3);
t3(r.xy, sin(z.x/10)/3);
t3(p.xz,z.x/3);
}
if (w==1 ) {
p=float3(-10,30,0);
t3(r.yz, 1.1);
t3(r.xz, z.x/4);
}
if (w==2 ) {
p=float3(25+sin(z.x/5)*3,10+3*sin(z.x/2),0);
t3(r.yz, sin(z.x/5)/2);
t3(r.xz, 1.54+z.x/5);
t3(r.xy, cos(z.x/10)/3);
t3(p.xz,z.x/5);
}
} else {
p=float3(0,4,8);
t3(r.yz,sin(z.x/5)/5);
t3(r.xy,cos(z.x/4)/2);
t3(r.xz,-1.54+smoothstep(0,4,z.x-155)*(z.x-155)/3);
}
// Boom effect on camera
p.x+=z.y*sin(111*z.x)/4;
// Lights
static float4 l[6] = {{.7,.2,0,2},{.7,0,0,3},{.02,.05,.2,7},
{(4+10*step(24,z.z))*cos(z.x/5),-5,(4+10*step(24,z.z))*sin(z.x/5),0},
{-30+5*sin(z.x/2),8,6+10*sin(z.x/2),0},
{25,25,10,0}
};
</pre>
<br />
<h3 id="statistics">
Compression statistics</h3>
<br />
Final compression results are given in the following table:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidE1BuXZoQIJZncpT-P_8jhcIzTff3pbPkNAGwjg84rjL8MrvrsqLBrXshwTSW-FJWdkf3PZV-rZQ0sX7TPqbWGJa7DYyqC9aULWF13bpFdmRF7HUqyyo8NVuCQ_5OaSZztwSAkBjrBZ4/s1600/ergon_compression_results.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidE1BuXZoQIJZncpT-P_8jhcIzTff3pbPkNAGwjg84rjL8MrvrsqLBrXshwTSW-FJWdkf3PZV-rZQ0sX7TPqbWGJa7DYyqC9aULWF13bpFdmRF7HUqyyo8NVuCQ_5OaSZztwSAkBjrBZ4/s640/ergon_compression_results.png" width="640" /></a></div>
<br />
So to summarize, total exe size is 4070 bytes, and is composed of :<br />
<ul>
<li>Synth code + music data is taking around 35% of the total exe size = 1461 bytes</li>
<li>Shader code is taking 36% = 1467 bytes</li>
<li>Main code + non shader data is 14% = 549 bytes</li>
<li>PE + crinkler decoder + crinkler import is 15% = 593 bytes</li>
</ul>
<br />
<br />
The intro was finished around the 13 march 2010, well ahead BP2010. So that was damn cool... I spent the rest of my time until BP2010 to try to develop a procedural 4k gfx, using D3D11 compute shaders, raymarching and a Global Illumination algorithm... but the results (algo finished during the party) disappointed me... And when I saw the fantastic <a href="http://pouet.net/prod.php?which=54544">Burj Babil by Psycho</a>, he was right about using a plain raymarcher without any complicated true light management... a good "basic" raymarching algo, with some tone mapping finetune was much more relevant here!<br />
<br />
Anyway, my GI experiment on the compute shader will probably deserve an article here.<br />
<br />
<hr />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoeHC4ymPh0JICWyzh4xtc7xEqIKhL1qxDLSp1mUHhszkqWc7Jt6xExymGu7QB4XXCI13kQ9SnaLpQBI94BHuXLlNmBJ84F1WZiBpYLI6ZqSIpZEjSFbIdWCd57ogi55zbM14qKAx3rVs/s1600/ergon+2010-03-11+23-35-46-40.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoeHC4ymPh0JICWyzh4xtc7xEqIKhL1qxDLSp1mUHhszkqWc7Jt6xExymGu7QB4XXCI13kQ9SnaLpQBI94BHuXLlNmBJ84F1WZiBpYLI6ZqSIpZEjSFbIdWCd57ogi55zbM14qKAx3rVs/s320/ergon+2010-03-11+23-35-46-40.jpg" /></a></div>
I really enjoyed to make this demo and to see that ergon was able to make it in the top 3... after seeing BP2009, I was not expecting at all the intro to be in the top 3!... although I know that the competition this year was far much easier than the previous BP!<br />
<br />
Anyway, that was nice to work with my friend ulrick... and to contribute to the demoscene with this prod. I hope that I will be able to keep on working on the demos like this... I still have lots of things to learn, and that's cool!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com5tag:blogger.com,1999:blog-1076643699683521890.post-69458037425522279732010-08-25T09:06:00.011+11:002010-08-25T10:26:36.298+11:00Import and Export 3D Collada files with C#/.NETLooking at what kind of 3D file format I could work with, I know that <a href="http://collada.org/">Collada </a>is a well established format, supported by several 3D modeling tools, with a public specification and a XML/Schema grammar description, very versatile - and thus very verbose. For the last years, I saw a couple of articles on, for example, "how to import them in the XNA content pipeline" or about <a href="http://abi.exdream.com/Blog/post/2007/02/25/Skeletal-Bone-Animation-and-Skinning-with-Collada-Models-in-XNA.aspx">Skinning Animation with Collada and XNA</a>, with some brute force code, using DOM or XPath to navigate around the Collada elements. <br />
<br />
Now, looking at the opportunity to use this format and to build a small 3D demo framework in C# around SlimDx, I tried to find a full implementation of a Collada loader, derived from the xsd official specification... but was disappointed to learn that most of the attempts failed to use the specification with an automatic tool like <a href="http://msdn.microsoft.com/fr-fr/library/x6c1kb0s%28VS.80%29.aspx">xsd.exe</a> from Microsoft. If you don't know what's xsd.exe, It's simply a tool to work with XML schemas, generate schemas from a DLL assembly, generate C# classes from a xsd schema...etc, very useful when you want to use directly from the code an object model described in xsd. I will explain later why this is more convenient to use it, and what you can do with it that you cannot achieve with the same efficiency compare to raw DOM/Xpath access.<br />
<br />
I had already used xsd tool in the past for <a href="http://nrenoisetools.codeplex.com/">NRenoiseTools </a>project and found it quite powerful and simple, and was finally quite happy with it... But why the Collada xsd was not working with this tool?<br />
<br />
<a name='more'></a><br />
<h3>Patching the Collada xsd</h3>Firstly, I have downloaded the <a href="http://www.khronos.org/collada/">Collada xsd spec</a> from Kronos group and ran it through the tool... too bad, there was indeed an error preventing xsd to work on it<br />
<code>Error: Error generating classes for schema 'COLLADASchema_141'.<br />
- Group 'glsl_param_type' from targetNamespace='http://www.collada.org/2005/11<br />
/COLLADASchema' has invalid definition: Circular group reference.</code><br />
This error was <a href="https://collada.org/public_forum/viewtopic.php?f=12&t=556&start=0">quite old</a> and got even a bug submitted to connect "<a href="http://connect.microsoft.com/VisualStudio/feedback/details/289668/xsd-exe-fails-with-collada-schema-prints-circular-reference-problem">xsd.exe fails with COLLADA schema. Prints circular reference problem</a>". Well the problem is that looking more deeply at the xsd schema, the glsl_param_type doesn't make any circular group reference... weird...<br />
<br />
Anyway, because this was just an error on the GLSL profile part of Collada spec, I removed this part, as this is not so much used... and did the same for CG and GLES profiles that had the same error.<br />
<br />
Bingo! Xsd.exe tool was able to generate a -large - C# source file. I found it so easy that I was wondering why they had so much pain with it in the past? Well, running a simple program to load a sample DAE collada files... and got a deep exception :<br />
<code><br />
Member 'Text' cannot be encoded using the XmlText attribute<br />
</code><br />
A few internet click away, I found exactly a guy <a href="http://social.msdn.microsoft.com/Forums/en-US/asmxandxml/thread/c515173f-b6dc-4e72-a08d-69b75382fe85">having the same error</a>... from the code:<br />
<pre class="brush: csharp" name="code">/// <remarks/>
[System.Xml.Serialization.XmlTextAttribute()]
public double[] Text {
get {
return this.textField;
}
set {
this.textField = value;
}
}
</pre>XmlTextAttribute specify that the "Text" property should be serialized inside the content of the xml element... but unfortunately, the XmlText attribute doesn't work on arrays of primitives!<br />
<br />
Someone suggested him several options, and the simplest among them was to use a simple string to serialize the content instead of using an array... This is a quite common trick if you are familiar with xml serializing in .NET (and also with WCF DataContract xml serialization from .NET). So I went this way... It was quite easy, because the file had less than 10 occurrences to patch, so I patched them manually... with the kind of following code:<br />
<pre class="brush: csharp" name="code">/// <remarks />
[XmlText]
public string _Text_
{
get { return COLLADA.ConvertFromArray(Values); }
set { Values = COLLADA.ConvertDoubleArray(value); }
}
/// <remarks />
[XmlIgnore]
public double[] Values
{
get { return textField; }
set { textField = value; }
}
</pre>I put a XmlIgnore on the renamed "Values" property that use the double[] and add a string property that performs a two-way conversion to that values (while adding the ConvertFromArray and ConvertDoubleArray functions at the end of the xsd generated file.<br />
<br />
And... It was fully working!<br />
<br />
<h3>Using Collada model from C#</h3>With the generated classes, this is much easier to safely read the document, to access collada elements, having intellisense completion to help you on this laborious task. I have also added just 2 methods to load and save directly dae files from a stream or a file. The code iterating on Collada elements is something like (dummy code):<br />
<pre class="brush: csharp" name="code">// Load the Collada model
COLLADA model = COLLADA.Load(inputFileName);
// Iterate on libraries
foreach (var item in model.Items)
{
var geometries = item as library_geometries;
if (geometries== null)
continue;
// Iterate on geomerty in library_geometries
foreach (var geom in geometries.geometry)
{
var mesh = geom.Item as mesh;
if (mesh == null)
continue;
// Dump source[] for geom
foreach (var source in mesh.source)
{
var float_array = source.Item as float_array;
if (float_array == null)
continue;
Console.Write("Geometry {0} source {1} : ",geom.id, source.id);
foreach (var mesh_source_value in float_array.Values)
Console.Write("{0} ",mesh_source_value);
Console.WriteLine();
}
// Dump Items[] for geom
foreach (var meshItem in mesh.Items)
{
if (meshItem is vertices)
{
var vertices = meshItem as vertices;
var inputs = vertices.input;
foreach (var input in inputs)
Console.WriteLine("\t Semantic {0} Source {1}", input.semantic, input.source);
}
else if (meshItem is triangles)
{
var triangles = meshItem as triangles;
var inputs = triangles.input;
foreach (var input in inputs)
Console.WriteLine("\t Semantic {0} Source {1} Offset {2}", input.semantic, input.source, input.offset);
Console.WriteLine("\t Indices {0}", triangles.p);
}
}
}
}
// Save the model
model.Save(inputFileName + ".test.dae");
</pre><br />
One thing that could be of an interest, is that not only you can easily load a Collada dae file... but <b>you can export them as well</b>! I did a couple of experiment to verify that importing and exporting a Collada file is producing the same file, and It seems to work like a charm... meaning that if you want to produce some procedural Collada models to load them back in a 3D modeling tool, this is quite straightforward! But anyway, my main concern was to have a solid Collada loader that is compliant with the spec and performs most of the tedious fields conversion for me.<br />
<br />
Of course, having such a loader in C# is just a very small part of the whole picture in order to create a full importer supporting most of the Collada features for a custom renderer... but that's probably the less exciting part of developing such an importer, so having this C# Collada model will be probably helpful.<br />
<br />
<hr><br />
Note: You can download the <a href="http://xoofx.com/upload/csharp_collada_schema_1_4.7z">C# Collada model here</a>. This is only a single C# source file that you can add directly to your project!<br />
<br />
The model is stored inside the namespace Collada141 (in order to support multiple incompatible version of the Collada spec), and the root class (as specified in the xsd) is the COLLADA class, which contains also the two added Load/Save methods.<br />
<br />
Also, a nice thing about the generated model from xsd.exe is that it allows you to extend the object model methods outside the csharp file. All the classes are declared partial, so It's quite easy to add some helpers method directly inside the Collada object model without touching directly the generated file.<br />
<br />
Let me know if you are using it!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com14tag:blogger.com,1999:blog-1076643699683521890.post-7432727959457648262010-08-13T07:58:00.005+11:002010-10-08T23:56:32.251+11:00Democoding, tools coding and coding scatteringNot so much post here for a while... So I'm going to just recap some of the coding work I have done so far... you will notice that It's going in lots of direction, depending on opportunities, ideas, sometimes not related to democoding at all... not really ideal when you want to release something! ;)<br />
<br />
So, here are some directions I have been working so far...<br />
<br />
<a name='more'></a><br />
<h3>C# and XNA</h3>I tried to work more with C#, XNA... looking for an opportunity to code a demo in C#... I even started a post about it few months ago, but leaving it in a draft state. XNA is really great, but I had some bad experience with it... I was able to use it without requiring a full install but while playing with model loading, I had a weird bug called the <a href="http://forums.xna.com/forums/p/52248/316364.aspx#316364">black model bug</a>. Anyway, I might come back to C# for DirectX stuff... SlimDx is for example really helpful for that.<br />
<br />
<h3>A 4k/64k softsynth</h3>I have coded a synth dedicated to 4k/64k coding. Although, right now, I only have the <b>VST and GUI fully working</b> under Renoise.. but not yet the asm 4k player! ;)<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj3L5QmhHRBxGJwJZxyz5iqJt6qKys0hdXZXCJH3r-gNxVqx9-jP51hdCNBSH2nNyIe83ku1tFyLYsEbDI6PpZj-DnyCtUUxnAUMeLAjE0df1gqvCcTtf6ZoIPGhfUucnQvdmSgja0LLA/s1600/helium4k.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj3L5QmhHRBxGJwJZxyz5iqJt6qKys0hdXZXCJH3r-gNxVqx9-jP51hdCNBSH2nNyIe83ku1tFyLYsEbDI6PpZj-DnyCtUUxnAUMeLAjE0df1gqvCcTtf6ZoIPGhfUucnQvdmSgja0LLA/s320/helium4k.png" /></a></div><br />
The main idea was to build a <b>FM8/DX7 like synth</b>, with exactly the same output quality (excluding some fancy stuff like the arpegiator...). The synth was developed in C# using <a href="http://vstnet.codeplex.com/">vstnet</a>, but must be more considered as a prototype under this language... because the asm code generated by the JIT is not really good when it comes to floating point calculation... anyway, It was really good to develop under this platform, being able to prototype the whole thing in few days (and of course, much more days to add rich GUI interaction!).<br />
<br />
I still have to add a sound library file manager and the <b>importer for DX7 patch</b>..... Yes, you have read it... my main concern is to provide as much as possible a tons of ready-to-use patches for ulrick (our musician at FRequency)... Decoding the DX7 patch is well known around the net... but the more complex part was to make it decode like the FM8 does... and that was tricky... Right now, every transform functions are in an excel spreadsheet, but I have to code it in C# now!<br />
<br />
You may wonder why developing the synth in C# if the main target is to code the player in x86 asm? Well, for practical reasons : I needed to quickly experiment the versatility of the sounds of this synth and I'm much more familiar with .NET winform to easily build some complex GUI. Although, I have done the whole synth with 4k limitation in mind... especially about data representation and complexity of the player routine.<br />
<br />
For example, for the 4k mode of this synth, waveforms are strictly restricted to only one : sin! No noise, no sawtooth, no square... what? A synth without those waveform?.... but yeah.... When I looked back at DX7 synth implem, I realized that they were using only a pure "sin"... but with the complex FM routing mechanism + the feedback on the operators, the DX7 is able to produce a large variety of sounds ranging from strings, bells, bass... to drumkits, and so on...<br />
<br />
I did also a couple of effects, mainly a versatile variable delay line to implement Chorus/Flanger/Reverb.<br />
<br />
So basically, I should end up with a synth with two modes :<br />
- <b>4k mode</b> : only 6 oscillators per instrument, only sin oscillators, simple ADSR envelope, full FM8 like routing for operators, fixed key scaling/velocity scaling/envelope scaling. Effects per instrument/global with a minimum delay line + optional filters. and last but not least, polyphony : that's probably the thing I miss the most in 4k synth nowadays...<br />
- <b>64k mode</b> : up to 8 oscillators per instrument, all FM8 oscillators+filters+WaveShaping+RingModulation operators, 64 steps FM8's like envelope, dynamic key scaling/velocity scaling/envelope scaling. More effects, with better quality, 2 effect //+serial line per instrument. Additional effects channel to route instrument to the same effects chain. Modulation matrix.<br />
<br />
The 4k mode is in fact restricting the use of the 64k mode, more at the GUI level. I'm currently targeting only the 4k mode, while designing the synth to make it ready to support 64k mode features.<br />
<br />
What's next? Well, finish the C# part (file manager and dx7 import) and starting the x86 asm player... I just hope to be under 700 compressed byte for the 4k player (while the 64k mode will be written in C++, with an easier limitation around 5Ko of compressed code) .... but hey, until It's not coded... It's pure speculation!.... And as you can see, the journey is far from finished! ;)<br />
<br />
<h3>Context modeling Compression update</h3>During this summer, I came back to my compression experiment I did last year... The current status is quite pending... The compressor is quite good, sometimes better than crinkler for 4k... but the prototype of the decompressor (not working, not tested....) is taking more than 100 byte than crinkler... So in the end, I know that I would be off more than 30 to 100 byte compared to crinkler... and this is not motivating me to finish the decompressor and to get it really running.<br />
<br />
The basic idea was to take the standard context modeling approach from <a href="http://mattmahoney.net/dc/">Matt Mahoney </a>(also known as <a href="http://en.wikipedia.org/wiki/PAQ">PAQ </a>compression, Matt did a fantastic job with his research, open source compressor....by the way), using dynamic neural network with an order of 8 (8 byte context history), with the same mask selection approach than crinkler + some new context filtering at the bit level... In the end, the decompressor is using the FPU to decode the whole thing... as it needs ln2() and pow2() functions... So during the summer, I though using another logistic activation function to get rid of the FPU : the standard sigmoid used in the neural network with a base 2 is 1/(1+2^-x)), so I found something similar with y = (x / (1 + |x|) + 1) /2 from David Elliot (some references <a href="http://www.dontveter.com/bpr/activate.html">here</a>). I didn't have any computer at this time to test it, so I spent few days to put some math optimization on it, while calculating the logit function (the inverse of this logistic function).<br />
<br />
I came back to home very excited to test this method... but I was really disappointed... the function had a very bad impact on compression ratio by a factor of 20%, in the end, completely useless!<br />
<br />
If by next year, I'm not able to release anything from this.... I will put all this work open source, at least for educational purposes... someone will certainly be clever than me on this and tweak the code size down!<br />
<br />
<h3>SlimDx DirectX wrapper's like in C++<br />
</h3>Recall that for the ergon intro, I have been working with a very thin layer around DirectX to wrap enums/interfaces/structures/functions. I did that around D3D10, a bit of D3D11, and a bit of D3D9 (which was the one I used for ergon). The goal was to achieve a DirectX C# like interface in C++. While the code has been coded almost entirely manually, I was wondering If I could not generate It directly from DirectX header files...<br />
<br />
So for the last few days, I have been a bit working on this... I'm using <a href="http://www.boost.org/doc/libs/1_43_0/libs/wave/doc/introduction.html">boost::wave </a>as the preprocessor library... and I have to admit that the C++ guy from boost lost their mind with templates... It's amazing how they did something simple so complex with templates... I wanted to use this under a C++/Cli managed .NET extension to ease my development in C#, but I end up with a template error at link stage... an incredible error with a line full of concatenated template, even freezing visual studio when I wanted to see the errors in the error list!<br />
<br />
Template are really nice, when they are used not too intensively... but when everything is templatized in your code, It's becoming very hard to use fluently a library and It's sometimes impossible to understand the template error, when this error is more than 100 lines full of cascading template types!<br />
<br />
Anyway, I was able to plug this boost::wave in a native dll, and calling it from a C# library... next step is to see how much I can get from DirectX header files to extract a form of IDL (Interface Definition Language). If I cannot get something relevant in the next week, I might postpone this task when I won't have anything more important to do! The good thing is for example for D3D11 headers, you can see that those files were auto-generated from a mysterious... d3d11.idl file...used internally at Microsoft (although It would have been easier to get directly this file!)... so It means that the whole header is quite easy to parse, as the syntax is quite systematic.<br />
<br />
Ok, this is probably not linked to intros... or probably only for 64k.... and I'm not sure I will be able to finish it (much like rmasm)... And this kind of work is keeping me away from directly working with DirectX, experimenting rendering techniques and so on... Well, I have to admit also that I have been more attracted for the past few years to do some tools to enhance coding productivity (not necessary only mine)... I don't like to do too much things manually.... so everytime there is an opportunity to automatize a process, I can't refrain me to make it automatic! :D<br />
<br />
<br />
<h3>AsmHighlighter and NShader next update<br />
</h3>Following my bad appetite for tools, I need to make some update to AsmHighlighter and NShader, to add some missing keywords, patch a bug, support for new VS2010 version... whatever... When you release this kind of open source project, well, you have to maintain them, even if you don't use them too much... because other people are using them, and are asking for improvements... that's the other side of the picture...<br />
<br />
So because I have to maintain those 2 projects, and they are in fact sharing logically more than 95% of the same code, I have decided to merge them into a single one... that will be available soon under codeplex as well. That will be easier to maintain, ending with only one project to update.<br />
<br />
<br />
The main features people are asking is to be able to <b>add some keywords easily </b>and to<b> map file extensions to the syntax highlighting system</b>... So I'm going to generalize the design of the two project to make them more configurable... hopefully, this will cover the main features request... <br />
<br />
<h3>An application for Windows Phone 7... meh?<br />
</h3>Yep... I have to admit that I'm really excited by the upcoming Windows Phone 7 metro interface... I'm quite fed up with my iPhone look and feel... and because the development environment is so easy with C#, I have decided to code an application for it. I'm starting with a chromatic tuner for guitar/piano/violins...etc. and it's working quite well, even if I was able to test it only under the emulator. While developing this application, I have learned some cool things about pitch detection algorithm and so on...<br />
<br />
I hope to finish the application around september, and to be able to test it with a real hardware when WP7 will be offcialy launched... and before puting this application on the windows marketplace.<br />
<br />
If this is working well, I would study to develop other applications, like porting the softsynth I did in C# to this platform... We will see... and definitely, this last part is completely unrelated to democoding!<br />
<br />
<br />
<h3>What's next?</h3>Well, I have to prioritize my work for the next months:<br />
<ol><li>Merge AsmHighlighter and NShader into a single project. </li>
<li>Play a bit for one week with DirectX headers to see if I could extract some IDL's like information</li>
<li>Finish the 4k mode of the softsynth... and develop the x86 asm player</li>
<li>Finish the WP7 application</li>
</ol>I still have also an article to write about ergon's making of, not much to say about it, but It could be interesting to write down on a paper those things....<br />
<br />
I need also to work on some new directX effects... I have played a bit with hardware instantiating, compute shaders (with a raymarching with global illumination for a 4k procedural compo that didn't make it to BP2010, because the results were not enough impressive, and too slow to calculate...)... I would really want to explore more about SSAO things with plain polygons... but I didn't take time for that... so yep, practicing more graphics coding should be on my top list... instead of all those time consuming and - sometimes useful - tools!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com2tag:blogger.com,1999:blog-1076643699683521890.post-81459974960591875492010-05-22T09:16:00.010+11:002010-08-13T00:16:52.620+11:00Playing a MP3 in c++ using plain Windows APIWhile playing a mp3 is quite common in a demo, I have seen that most demo are often using 3rd party dlls like <a href="http://www.un4seen.com/">Bass </a>or <a href="http://www.fmod.org/">FMod</a> to perform this simple task under windows. But if we want to get rid off this dependency, how can we achieve this with a plain windows API? What's the requirements to have a practical MP3Player for a demo?<br />
<br />
Surprisingly, I was not able to find a simple code sample other the Internet that explain how to play a mp3 with Windows API, without using the too simple <a href="http://msdn.microsoft.com/en-us/library/dd758070%28VS.85%29.aspx">Windows Media Player API</a>. Why WMP is not enough (not even talking about MCI - Media Control Interface which is even more basic than WMP)?<br />
<br />
Well, It's lacking one important feature : It's only able to play from an url, so it's not possible for example to pack the song in an archive and play it from a memory location (although not a huge deal if you want to release the song on the side of your demo). Also I have never tested the timing returned by WMP (using probably <a href="http://msdn.microsoft.com/en-us/library/dd563183%28v=VS.85%29.aspx">IWMPControls3 getPositionTimeCode</a>) and not really sure It's able to provide a reliable sync (at least, If you intend to use sync... but hey, is a demo without any sync, can be still called a demo?:)<br />
<br />
<a name='more'></a>So I started to find some peace of code around the net but they were covering only part of the problem. The starting point was to rely on the <a href="http://msdn.microsoft.com/en-us/library/dd797805%28v=VS.85%29.aspx">Audio Compression Manager API</a> that provides an API conversion to perform for example a mp3 to pcm. Hopefully, I found the code from a guy that was kind enough to post <a href="http://david.weekly.org/code/mp3acm.html">the whole converter for a mp3 file using ACM</a>. In the mean time, I found that Mark Heath, the author of NAudio posted few days ago a solution to <a href="http://mark-dot-net.blogspot.com/2010/05/converting-mp3-to-wav-with-naudio.html">convert a MP3 to WAV using NAudio</a>. Looking at his code, he was using also ACM but he reported also some difficulties to implement a reliable MP3Frame/ID3Tag decoder in order to extract samplerate, bitrate, channels...etc. I didn't want to use this kind of heavy code and was looking a lighter and reliable solution for this : most of the people were talking about using the <a href="http://msdn.microsoft.com/en-us/library/dd757738%28v=VS.85%29.aspx">Windows Media Format SDK</a> to get all this information from the file. The starting point is the WMCreateSyncReader method. Through this method, you are able to retrieve part of MP3Frame as well as ID3Tag.<br />
<br />
Finally, I came with a patchwork solution :<br />
<ul><li>using SyncReader from WMF to extract song duration.</li>
<li>using ACM to decode the mp3 to pcm</li>
<li>using plain old waveOut functions to perform sound playback and retrieve sound playback position.</li>
</ul>Everything is inside a single .h with less than 300 lines including comments. I don't really know If it's the best way to play a mp3 from a file or from the memory, with Windows API, while still providing a reliable timing. I have tested it against a couple of mp3, thus It may still have some bugs... but at least, It's working quite well and It's a pretty small code. For example, the code is expecting the input mp3 to be at 44100Hz samplerate... If not It should probably failed... although with the use of WMF, It's quite easy to extract the sampleRate (although I'm not using it in the sample code provided here... was not sure about the result though :) )<br />
<br />
Also, the code is not decoding&playing in realtime the song but is instead performing the decoding in a single pass and then playing the decoded buffer. This requires that the full pcm song to be allocated, which could be around 20Mo to 50Mo depending on the size of your song (It's easy to calculate : durationInSecondsOfTheSong * 4 * 441000, so a 3min song is requiring 30Mo). This is not probably the best solution, but It's not a huge task to transform this code to do realtime decoding/playback. The downside is that It will take some CPU in your demo. So that in the end, It's a just tradeoff between memory vs cpu depending on your needs!<br />
<br />
<pre class="brush: cpp" name="code">/* ----------------------------------------------------------------------
* MP3Player.h C++ class using plain Windows API
*
* Author: @lx/Alexandre Mutel, blog: http://code4k.blogspot.com
* The software is provided "as is", without warranty of any kind.
* ----------------------------------------------------------------------*/
#pragma once
#include <windows.h>
#include <stdio.h>
#include <assert.h>
#include <mmreg.h>
#include <msacm.h>
#include <wmsdk.h>
#pragma comment(lib, "msacm32.lib")
#pragma comment(lib, "wmvcore.lib")
#pragma comment(lib, "winmm.lib")
#pragma intrinsic(memset,memcpy,memcmp)
#ifdef _DEBUG
#define mp3Assert(function) assert((function) == 0)
#else
//#define mp3Assert(function) if ( (function) != 0 ) { MessageBoxA(NULL,"Error in [ " #function "]", "Error",MB_OK); ExitProcess(0); }
#define mp3Assert(function) (function)
#endif
/*
* MP3Player class.
* Usage :
* MP3Player player;
* player.OpenFromFile("your.mp3");
* player.Play();
* Sleep((DWORD)(player.GetDuration()+1));
* player.Close();
*/
class MP3Player {
private:
HWAVEOUT hWaveOut;
DWORD bufferLength;
double durationInSecond;
BYTE* soundBuffer;
public:
/*
* OpenFromFile : loads a MP3 file and convert it internaly to a PCM format, ready for sound playback.
*/
HRESULT OpenFromFile(TCHAR* inputFileName){
// Open the mp3 file
HANDLE hFile = CreateFile(inputFileName, // open MYFILE.TXT
GENERIC_READ,
FILE_SHARE_READ, // share for reading
NULL, // no security
OPEN_EXISTING, // existing file only
FILE_ATTRIBUTE_NORMAL, // normal file
NULL); // no attr
assert( hFile != INVALID_HANDLE_VALUE);
// Get FileSize
DWORD fileSize = GetFileSize(hFile, NULL);
assert( fileSize != INVALID_FILE_SIZE);
// Alloc buffer for file
BYTE* mp3Buffer = (BYTE*)LocalAlloc(LPTR, fileSize);
// Read file and fill mp3Buffer
DWORD bytesRead;
DWORD resultReadFile = ReadFile( hFile, mp3Buffer, fileSize, &bytesRead, NULL);
assert(resultReadFile != 0);
assert( bytesRead == fileSize);
// Close File
CloseHandle(hFile);
// Open and convert MP3
HRESULT hr = OpenFromMemory(mp3Buffer, fileSize);
// Free mp3Buffer
LocalFree(mp3Buffer);
return hr;
}
/*
* OpenFromMemory : loads a MP3 from memory and convert it internaly to a PCM format, ready for sound playback.
*/
HRESULT OpenFromMemory(BYTE* mp3InputBuffer, DWORD mp3InputBufferSize){
IWMSyncReader* wmSyncReader;
IWMHeaderInfo* wmHeaderInfo;
IWMProfile* wmProfile;
IWMStreamConfig* wmStreamConfig;
IWMMediaProps* wmMediaProperties;
WORD wmStreamNum = 0;
WMT_ATTR_DATATYPE wmAttrDataType;
DWORD durationInSecondInt;
QWORD durationInNano;
DWORD sizeMediaType;
DWORD maxFormatSize = 0;
HACMSTREAM acmMp3stream = NULL;
HGLOBAL mp3HGlobal;
IStream* mp3Stream;
// Define output format
static WAVEFORMATEX pcmFormat = {
WAVE_FORMAT_PCM, // WORD wFormatTag; /* format type */
2, // WORD nChannels; /* number of channels (i.e. mono, stereo...) */
44100, // DWORD nSamplesPerSec; /* sample rate */
4 * 44100, // DWORD nAvgBytesPerSec; /* for buffer estimation */
4, // WORD nBlockAlign; /* block size of data */
16, // WORD wBitsPerSample; /* number of bits per sample of mono data */
0, // WORD cbSize; /* the count in bytes of the size of */
};
const DWORD MP3_BLOCK_SIZE = 522;
// Define input format
static MPEGLAYER3WAVEFORMAT mp3Format = {
{
WAVE_FORMAT_MPEGLAYER3, // WORD wFormatTag; /* format type */
2, // WORD nChannels; /* number of channels (i.e. mono, stereo...) */
44100, // DWORD nSamplesPerSec; /* sample rate */
128 * (1024 / 8), // DWORD nAvgBytesPerSec; not really used but must be one of 64, 96, 112, 128, 160kbps
1, // WORD nBlockAlign; /* block size of data */
0, // WORD wBitsPerSample; /* number of bits per sample of mono data */
MPEGLAYER3_WFX_EXTRA_BYTES, // WORD cbSize;
},
MPEGLAYER3_ID_MPEG, // WORD wID;
MPEGLAYER3_FLAG_PADDING_OFF, // DWORD fdwFlags;
MP3_BLOCK_SIZE, // WORD nBlockSize;
1, // WORD nFramesPerBlock;
1393, // WORD nCodecDelay;
};
// -----------------------------------------------------------------------------------
// Extract and verify mp3 info : duration, type = mp3, sampleRate = 44100, channels = 2
// -----------------------------------------------------------------------------------
// Initialize COM
CoInitialize(0);
// Create SyncReader
mp3Assert( WMCreateSyncReader( NULL, WMT_RIGHT_PLAYBACK , &wmSyncReader ) );
// Alloc With global and create IStream
mp3HGlobal = GlobalAlloc(GPTR, mp3InputBufferSize);
assert(mp3HGlobal != 0);
void* mp3HGlobalBuffer = GlobalLock(mp3HGlobal);
memcpy(mp3HGlobalBuffer, mp3InputBuffer, mp3InputBufferSize);
GlobalUnlock(mp3HGlobal);
mp3Assert( CreateStreamOnHGlobal(mp3HGlobal, FALSE, &mp3Stream) );
// Open MP3 Stream
mp3Assert( wmSyncReader->OpenStream(mp3Stream) );
// Get HeaderInfo interface
mp3Assert( wmSyncReader->QueryInterface(&wmHeaderInfo) );
// Retrieve mp3 song duration in seconds
WORD lengthDataType = sizeof(QWORD);
mp3Assert( wmHeaderInfo->GetAttributeByName(&wmStreamNum, L"Duration", &wmAttrDataType, (BYTE*)&durationInNano, &lengthDataType ) );
durationInSecond = ((double)durationInNano)/10000000.0;
durationInSecondInt = (int)(durationInNano/10000000)+1;
// Sequence of call to get the MediaType
// WAVEFORMATEX for mp3 can then be extract from MediaType
mp3Assert( wmSyncReader->QueryInterface(&wmProfile) );
mp3Assert( wmProfile->GetStream(0, &wmStreamConfig) );
mp3Assert( wmStreamConfig->QueryInterface(&wmMediaProperties) );
// Retrieve sizeof MediaType
mp3Assert( wmMediaProperties->GetMediaType(NULL, &sizeMediaType) );
// Retrieve MediaType
WM_MEDIA_TYPE* mediaType = (WM_MEDIA_TYPE*)LocalAlloc(LPTR,sizeMediaType);
mp3Assert( wmMediaProperties->GetMediaType(mediaType, &sizeMediaType) );
// Check that MediaType is audio
assert(mediaType->majortype == WMMEDIATYPE_Audio);
// assert(mediaType->pbFormat == WMFORMAT_WaveFormatEx);
// Check that input is mp3
WAVEFORMATEX* inputFormat = (WAVEFORMATEX*)mediaType->pbFormat;
assert( inputFormat->wFormatTag == WAVE_FORMAT_MPEGLAYER3);
assert( inputFormat->nSamplesPerSec == 44100);
assert( inputFormat->nChannels == 2);
// Release COM interface
// wmSyncReader->Close();
wmMediaProperties->Release();
wmStreamConfig->Release();
wmProfile->Release();
wmHeaderInfo->Release();
wmSyncReader->Release();
// Free allocated mem
LocalFree(mediaType);
// -----------------------------------------------------------------------------------
// Convert mp3 to pcm using acm driver
// The following code is mainly inspired from http://david.weekly.org/code/mp3acm.html
// -----------------------------------------------------------------------------------
// Get maximum FormatSize for all acm
mp3Assert( acmMetrics( NULL, ACM_METRIC_MAX_SIZE_FORMAT, &maxFormatSize ) );
// Allocate PCM output sound buffer
bufferLength = durationInSecond * pcmFormat.nAvgBytesPerSec;
soundBuffer = (BYTE*)LocalAlloc(LPTR, bufferLength);
acmMp3stream = NULL;
switch ( acmStreamOpen( &acmMp3stream, // Open an ACM conversion stream
NULL, // Query all ACM drivers
(LPWAVEFORMATEX)&mp3Format, // input format : mp3
&pcmFormat, // output format : pcm
NULL, // No filters
0, // No async callback
0, // No data for callback
0 // No flags
)
) {
case MMSYSERR_NOERROR:
break; // success!
case MMSYSERR_INVALPARAM:
assert( !"Invalid parameters passed to acmStreamOpen" );
return E_FAIL;
case ACMERR_NOTPOSSIBLE:
assert( !"No ACM filter found capable of decoding MP3" );
return E_FAIL;
default:
assert( !"Some error opening ACM decoding stream!" );
return E_FAIL;
}
// Determine output decompressed buffer size
unsigned long rawbufsize = 0;
mp3Assert( acmStreamSize( acmMp3stream, MP3_BLOCK_SIZE, &rawbufsize, ACM_STREAMSIZEF_SOURCE ) );
assert( rawbufsize > 0 );
// allocate our I/O buffers
static BYTE mp3BlockBuffer[MP3_BLOCK_SIZE];
//LPBYTE mp3BlockBuffer = (LPBYTE) LocalAlloc( LPTR, MP3_BLOCK_SIZE );
LPBYTE rawbuf = (LPBYTE) LocalAlloc( LPTR, rawbufsize );
// prepare the decoder
static ACMSTREAMHEADER mp3streamHead;
// memset( &mp3streamHead, 0, sizeof(ACMSTREAMHEADER ) );
mp3streamHead.cbStruct = sizeof(ACMSTREAMHEADER );
mp3streamHead.pbSrc = mp3BlockBuffer;
mp3streamHead.cbSrcLength = MP3_BLOCK_SIZE;
mp3streamHead.pbDst = rawbuf;
mp3streamHead.cbDstLength = rawbufsize;
mp3Assert( acmStreamPrepareHeader( acmMp3stream, &mp3streamHead, 0 ) );
BYTE* currentOutput = soundBuffer;
DWORD totalDecompressedSize = 0;
static ULARGE_INTEGER newPosition;
static LARGE_INTEGER seekValue;
mp3Assert( mp3Stream->Seek(seekValue, STREAM_SEEK_SET, &newPosition) );
while(1) {
// suck in some MP3 data
ULONG count;
mp3Assert( mp3Stream->Read(mp3BlockBuffer, MP3_BLOCK_SIZE, &count) );
if( count != MP3_BLOCK_SIZE )
break;
// convert the data
mp3Assert( acmStreamConvert( acmMp3stream, &mp3streamHead, ACM_STREAMCONVERTF_BLOCKALIGN ) );
// write the decoded PCM to disk
//count = fwrite( rawbuf, 1, mp3streamHead.cbDstLengthUsed, fpOut );
memcpy(currentOutput, rawbuf, mp3streamHead.cbDstLengthUsed);
totalDecompressedSize += mp3streamHead.cbDstLengthUsed;
currentOutput += mp3streamHead.cbDstLengthUsed;
};
mp3Assert( acmStreamUnprepareHeader( acmMp3stream, &mp3streamHead, 0 ) );
LocalFree(rawbuf);
mp3Assert( acmStreamClose( acmMp3stream, 0 ) );
// Release allocated memory
mp3Stream->Release();
GlobalFree(mp3HGlobal);
return S_OK;
}
/*
* Close : close the current MP3Player, stop playback and free allocated memory
*/
void __inline Close() {
// Reset before close (otherwise, waveOutClose will not work on playing buffer)
waveOutReset(hWaveOut);
// Close the waveOut
waveOutClose(hWaveOut);
// Free allocated memory
LocalFree(soundBuffer);
}
/*
* GetDuration : return the music duration in seconds
*/
double __inline GetDuration() {
return durationInSecond;
}
/*
* GetPosition : return the current position from the sound playback (used from sync)
*/
double GetPosition() {
static MMTIME MMTime = { TIME_SAMPLES, 0};
waveOutGetPosition(hWaveOut, &MMTime, sizeof(MMTIME));
return ((double)MMTime.u.sample)/( 44100.0);
}
/*
* Play : play the previously opened mp3
*/
void Play() {
static WAVEHDR WaveHDR = { (LPSTR)soundBuffer, bufferLength };
// Define output format
static WAVEFORMATEX pcmFormat = {
WAVE_FORMAT_PCM, // WORD wFormatTag; /* format type */
2, // WORD nChannels; /* number of channels (i.e. mono, stereo...) */
44100, // DWORD nSamplesPerSec; /* sample rate */
4 * 44100, // DWORD nAvgBytesPerSec; /* for buffer estimation */
4, // WORD nBlockAlign; /* block size of data */
16, // WORD wBitsPerSample; /* number of bits per sample of mono data */
0, // WORD cbSize; /* the count in bytes of the size of */
};
mp3Assert( waveOutOpen( &hWaveOut, WAVE_MAPPER, &pcmFormat, NULL, 0, CALLBACK_NULL ) );
mp3Assert( waveOutPrepareHeader( hWaveOut, &WaveHDR, sizeof(WaveHDR) ) );
mp3Assert( waveOutWrite ( hWaveOut, &WaveHDR, sizeof(WaveHDR) ) );
}
};
#pragma function(memset,memcpy,memcmp)
</pre><br />
The usage is then pretty simple :<br />
<br />
<pre class="brush: cpp" name="code">MP3Player player;
// Open the mp3 from a file...
player.OpenFromFile("your.mp3");
// or From a memory location!
player.OpenFromMemory(ptrToMP3Song, bytesLength);
player.Play();
while (...) {
// Do here your synchro in the demo using
...
double playerPositionInSeconds = player.GetPosition()
...
}
player.Close();
</pre><br />
And that's all! Hope someone will find this useful! <br />
<br />
You can download a <a href="http://xoofx.com/upload/TestMP3Player.zip">Visual Studio project using the MP3Player.h class</a>.xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com15tag:blogger.com,1999:blog-1076643699683521890.post-41437321153525116702010-05-20T20:49:00.004+11:002010-05-20T21:40:39.984+11:00NShader 1.1, hlsl, glsl, cg syntax coloring for Visual Studio 2008 & 2010I have recently released <a href="http://nshader.codeplex.com/">NShader</a> 1.1 which adds support for Visual Studio 2010 as well as bugfixes for hlsl/glsl syntax highlighting.<br />
<br />
While this plugin is quite cool to add a basic syntax highlighting for shader languages, It lacks intellisense/completion/error markers to improve the editor experience. I didn't have time to add such a functionality in this release as... I don't really have too much time dedicated to this project... and well, I have so much to learn from effectively practicing a lot more shader languages that I'm fine with this basic syntax highlighting! ;) Is it a huge task to add intellisense? It depends, but concretely, I need to implement for each shading language a full grammar/lexer parser in order to provide a reliable intellisense. Of course, a very basic intellisense would be feasible without this, but I would rather not to use an annoying/unreliable intellisense popup.<br />
<a name='more'></a><br />
Although, I did some research about existing lexers for shading languages, surprisingly, this is not something you can find easily. For hlsl for example, afaik, there is no bnf grammar published by Microsoft, so If you want to do it yourself, you need to go through the whole HLSL reference documentation and compile yourself a bnf... and that's something I can't afford in my spare time. One could argue that there are some startup code available on the net (<a href="http://src.chromium.org/svn/trunk/src/o3d/compiler/technique/Technique.g3pl">O3D from google</a> has an antlr parser/lexer, or a relative simpler <a href="http://code.google.com/p/schladetsch/source/browse/trunk/Effects/Tools/EffectTranslator/Grammars/Shader.g">one from Christian Schladetsch</a>), agree with that, but well... It still ask a bit more time to patch them, add support for SM5.0, handle correctly preprocessor directives... and so on... After that, I need to integrate it through the language service API, not the worst part. Anyway, If someone is motivated to help me on this, we could come with something. We will follow also if <a href="http://forums.xna.com/forums/p/7908/318939.aspx">Intelishade is able to resurrect in an open source way</a>... a joint venture would be interesting. <br />
<br />
Also, what's my feedback about migrating VS2008 language service to VS2010? Well, It was pretty straightforward! I did follow the sdk instructions about "<a href="http://msdn.microsoft.com/en-us/library/dd885475%28VS.100%29.aspx">Migrating a Legacy Language Service</a>" but It was not fully working as expected. In fact, the only remaining problem was that the <a href="http://social.msdn.microsoft.com/Forums/en-US/vsx/thread/b7ff2dfb-a391-46ee-9f52-862b9b4c9e77">WSIX VS2001 installer didn't register automatically the NShader Language Service</a>. I was forced to add manually the pkgdef file (containing registry update for the language service) to the vsix archive. While I was working on the migration to VS2010, I had a look at the new <a href="http://msdn.microsoft.com/en-us/library/dd885242.aspx">extensibility framework </a>and was surprised to see that the new framework is by far much easier to implement in VS2010. Although, I didn't take the time to migrate NShader to use this new framework, It seems to be pretty easy... also nice thing is that they did provide a compatibility layer for legacy Language Service, so I didn't bother with the new api. But If I had to write a new plugin for VS, I would definitely use the new API, although It would only work with VS2010+ versions...<br />
<br />
One small recurrent disappointment : Visual Studio is still restricting to provide plugins for Express editions. From a "commercial point of view", I understand this restriction, although for the thousands (million maybe?) of people using express edition, this is a huge lack of functionality.I'm sure that allowing community plugins into Express Editions would in fact improve a lot more Visual Studio adoption.<br />
<br />
My next post should be about the making of <a href="http://pouet.net/prod.php?which=54529">Ergon at BP2010</a>. I have a couple of things to share about it, but I'm quite lazy at that time to write this post... but It's on the way! ;)xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com3tag:blogger.com,1999:blog-1076643699683521890.post-51908664715490965232010-03-09T09:38:00.001+11:002010-05-20T21:41:40.347+11:00Work in progress, 4k intro for Breakpoint 2010Hey, almost one month that I have posted a message about coding a 4k intro for BP 2010. So what's going on?<br />
<br />
Well, I'm happy to be close to the end, fighting with the last available bytes, working on the music with ulrick, running, and rerunning several times the intro to work on the synchro. We are almost done now, but It's far from what I was expecting to release! I have developed around 4 to 5 scenes but was forced to only use 2 of them. I will publish a complete detail about compression ratio/vs part of this intro after releasing it. <br />
<br />
We have finally worked with 4klang for the music. This synth is very well designed and very versatile. The only drawback would be the total size code + data that goes around 1.4k to 1.5k. That's a lot and It didn't help me to inject more scenes. I would expect a synth code + data to be around 1k to 1.1k max, of course with less flexibility and probably a sound that could not be as rich as 4klang, but still with something nice. I have started to implement a small softsynth player based on my previous work. But I have suspended this laborious task, as I know that It would have taken me much more time to plug everything in a VST (although It's quite easy when you do this through .NET), design a simple GUI, file formats, develop a cool sound bank, test it, debug it....etc. That option was too risky, so I have postponed this work after BP.<br />
<br />
Anyway, from what I have seen while starting coding this softsynth, is that It's possible to go around 600 byte for the player... and probably 400 byte for the music + the sound bank... but I will be able to confirm this when I'm done with it, It's only a projection. The fact is that a stackbased synth is powerful, allowing for complex sound design (and synchronization/modulation, nicely done in 4klang) but If you look closer at it, you will see that most of the synth part are almost common to all sounds : While a stackbased synth allows everything, pragmatically, a sound respond to some classic design rules : a collection of cascaded/combined oscillators/noisers, few insert-fx (and the most common ones, filters, stereo delay/reverb), and those rules have a straight translation to a stack based system that you will recognize immediately among different instruments (and this pattern will repeat). This "static" pattern is probably more efficient for a 4k, both for the code size and the data, allowing for example to "bitify" data organization. Crinkler is using exactly this kind of static pattern (I still do have to write an article about it) : instead of having a generic context modeling compressor, It's using some kind of semi-dynamic/static context modeling that is in the end, much more powerful then it's generic equivalent (from what I have seen, a generic context modeling decompressor code is around 2% to 4% larger for a 4k, not a huge deal, but 2 to 4% is around 80 to 160 bytes for a 4k.... and that's a lot for a 4k). Of course, this is not only a question of some static algo vs more generic one... it's also a question of being able to produce a well done code-size optimized x86 assembler code (and crinkler just for this, is a masterpiece).<br />
<br />
So, 400 bytes would have allowed me to add a scene, add some nice text... :) But ok, we had to move on! I'm not going to complain about 4klang, when rudebox was so impressive, using exactly the same synth. It's possible to code something cool... and in the mean time to get some benefits from 4klang sound quality, so I found this an acceptable compromise... and a nice challenge!<br />
<br />
Anyway, see you soon at BP 2010!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com2tag:blogger.com,1999:blog-1076643699683521890.post-31761016361279495822010-02-13T02:12:00.001+11:002010-02-13T08:24:19.692+11:00Coding a 4k intro for Breakpoint 2010I'm going to be quite busy here until the demoparty <a href="http://breakpoint.untergrund.net/">Breakpoint 2010</a>, as I want to release a small 4k intro for this great event! I didn't plan anything few weeks ago and was slowly working on d3d10/d3d11, working on some effects for a 16k/64k intro to release later this year... but BP2010 is supposed to be the last one and I don't want to miss this experience, first time to go there... and probably last time, so I have decided to go to this party and try to contribute to it, yep!<br />
<br />
I have started to work with d3d10, as i wanted to add some direct2D nice layout text over 2-3 raymarching scenes... but I have found that direct2D is bit too costly for a 4k, specially if you want to avoid the basic white logo... so I have to switch back to a plain d3d9 and forget about some cool text. I will use this direct2D technique for a bigger intro. Due to my lack of investment in all the d3d apis until now, I had to manage the API from the ground up... and it took me a while... To facilitate the d3d10/d3d11 coding experience, I have developed <b>a lightweight c++ wrapper around d3d10/11 APIs, almost exactly in the same way (naming conventions,enums) <a href="http://slimdx.org/">SlimDx </a>has done the job</b>, and I'm really happy with it. The d3d10/d3d11 API is very clean but due to some verbose API contraints (ugly enums mixed sometimes with some #define, HRESULT to check from every function return, a famous windows programming philosophy), It's really worth to wrap this API around something that transparently hide all those things, rename and rearrange enum/methods/interface. Currently, It's working great with this SlimDx's like wrapper, much easier to program, much easier to read, and It does generate almost exactly the same code than a straight d3d10/d3d11 code, with the capability to remove the HRESULT check and so on... I will probably release this wrapper ( a single .h with a bunch of inline methods) around codeplex later this year, as it should probably help people like me that prefer a syntax much more closer to the C# coding experience than c/c++.<br />
<br />
During my small d3d11 incursion, I have also discovered that <b>the DirectX Effect framework (fx syntax files, with techniques, pass) is no longer available as part of the D3DX runtime!</b> Yep, you need to go to the Utility directory in the DirectX SDK to find that you can compile this framework yourself... It means that for all intros, you can forget about the DirectX Effect framework and program a much lighter Effect framework. One good thing about this change is that I had to better understand the d3d10 philosophy with the constant buffers access and so on... In fact, It's much easier to work directly with constant buffer, and surprisingly, It gives a smaller code. As soon as I'm done with the 4k intro for BP2010, I will publish a small post about this along the SlimDx like wrapper.<br />
<br />
Last thing is that I didn't have time to finish my softsynth, because I was more targeting a 16k/64k, with a more complex synth... so we should go with the great <a href="http://www.blogger.com/goog_1265984348575">4klang </a><a href="http://4klang.untergrund.net/">gopher's synth</a>... but I have two problems with it : 1) ulrick (my old friend, main FRequency musician) is unable to use it under Renoise. It burns its notebook's CPU and we don't know why, as it's a pretty standard core 2 duo intel processor... we did check lost of 4klang/renoise/system parameters without any success. 2) 4klang is great, but, the total code is often close to 900 bytes... I know that some of the top's 4k softsynth are around 500 to 700 bytes, so I'm not completely sure that I will use 4klang. The other idea is to take part of the work I did already for the bigger synth (that is developped in assembler x86), and try to plumb it into a fixed pipeline (and not a stackbased as 4klang)... I'm not sure, but I suspect that I could save a substantial amount of bytes... but still, not sure, and I need to check it... but It's going to be really hard to make it on time for BP... so we'll see...<br />
<br />
Currently, I have only coded one scene for the 4k intro, I'm quite happy with it... but that doesn't make a full intro! I need to add at least 3 scenes, work on the transitions, overall design, synch with synth & so on... even for a 4k, that's a lots of work, moreover when you consider that this is my first prod on my own (I mean, first prod for the PC, after the 3 prods i released 20 years ago on Amiga! ;) ) but It's possible in less than 2 month to do it, so I'll try to do my best!xoofx - Alexandre MUTELhttp://www.blogger.com/profile/05096938106073079832noreply@blogger.com1