Coding a 4k or 64K intro needs tools!

What makes a 4k or a 64k so special compares to others demo categories? Specialized tools of course!

There are some obvious general remarks we may list:

• The smaller the intro, the more specialized tools you will need (eg. crinkler, 4klang...). The bigger the demo (>5Mo), the more you will be able to use standardized tools (Zip, Blender, Renoise, Fmod...).
  
• The smaller the intro (<1k), the less specialized tools are released or even developed.
• The smaller the intro, the more graphists are being excluded from demo development compare to musicians that can still work on a music even in a 1k intro (see Untraceable).

Although this is not a prerequisite for coding a top notch intro, you may have noticed that most of the 4k-64k intros are often using some tools during their development (check out for example "The Making of Candystall", an interesting article describing the amount of work on tools for a 4k...). From higher order of relevance to lower, the 5 top tools for 4k-64k intros would probably be:

1. Cruncher, compactors, compressors : crinkler  (for 4k intros) or kkrunchy (for 64k) are widely used to crunch an intro. I'm convinced that this is the place to start for coding a tool for an intro. Even if crinkler is still the reference for 4k intros, i have investigated a little bit compression methods and worked on a small compressor, and i have found that it's still possible to increase compression ratio for intros in the range 4k-64k. Under 4k, it is almost impossible, as we are close to entropy (and well, crinkler's code decompressor is really impressive), but could be relevant in some cases. But above 4k, there is still plenty of room for improvement! I'm expecting to use and release our own compressor for 4k-64k that will be available open source (well, once we have used it a little bit! ;) ). An article will follow about this. Not to mention that for 1k category, there are no specialized tools available : for example, the untraceable code compressor from mentor is a custom derived work from crinkler's technique (and the compressor code, like crinkler, is impressive), or hitchhikr have released a nice hack with 1kpack, but that's all.
   
2. Softsynth, small synth trackers : 4klang (for 4k intros) or the v2 synthesizer (for 64k) are also widely used to develop music for intros. This is the 2nd place to start when coding tools for intros. Coding a softsynth is really funny and probably one of the most interesting part of a 4k. There will be a separate post (and probably severals) on that too, as it is a critical intro component. This is also often the place to integrate music-graphics synchronization, although some other specialized tools can also be used (like GNU rocket that was used by mentor or blueberry in somes intros, but I don't remember which one!)
   
3. Shader GLSL/HLSL code packer : For 4k, this is definitely the 3rd tools to have. Although the packing can be performed manually, a specialized tool is often used. I'm not sure there are any public tools released, but this is one of the easiest tools to develop so...
4. Procedurally Texture / Geometry / Path camera generators : it may not be used in 4k but this is often used for 64k. This is probably the 4th tools to have. Lots of groups have developed their own framework to perform this generation. Most notably, far-brausch with their ".werkkzeug 3 TE",  used in great productions like fr-041: debris. Although, iq has noticed that "procedurally textures" are not so common in 4k, mostly because this is not the 1st thing that makes your 4k intro nice... With todays SphereTracing technique (even if they are often using some kind of procedurally generated perlin's like texture) for 4k, textures are not really common in 4k but are still relevant in 64k.
5. Intro Studios : tools to visually edit your intros with timeframe, links to texture, 3D objects... like "werkkzeug1".  I'm not really a pure fan of that kind of tools but they are around and quite used in 64k intros (from Farbrausch to Conspiracy). 

Starting from 2007, i have mainly spent my time developing some of those tools (or "helloworld" tools to check "feasibility"). You may have noticed some open source projects that i have released so far. Some of them are being used as components of a demotools chain :

• NRenoiseTools was developed last year, mainly to perform a direct translation of xrns songs module to an internal synth trackers format. Others demomakers are often using a direct recorder with quantization inside their VST (see 4klang or V2) but i found that extracting the song directly from the song track format file can be much more efficient to find patterns repetitions and build optimized songs. I will post an article about this.
• NShader was recently developed to add syntax highlighting in Visual Studio for HLSL, GLSL shading languages. This project is also used to perform HLSL/GLSL code packing, although it can be developped also with a simple program using regular expression pattern matching (i used this technique for a 1st version but found lex tokenizer to be more precise).
• NKnobMan was also recently developed. It's not strictly a demotool, but i'm using it to speed up the creation of "template knobs" for our future 4k-64k synth VST.

NAudio was my first open source collaboration, working with Mark Heath, the creator and principal maintainer of this project, that provides native audio output under .NET. I have worked on this because when i came back to the demoscene two years ago, i was thinking about developing demos under .NET, but well, i have discovered more interesting things to do in the 4k-64k intros categories! You may be surprised, but .NET doesn't have any audio built-in capabilities and NAudio is the only project i know so far that provides this.

NetAsm was also a tentative to bring x86 ASM code injector for .NET. It was very promising, but Microsoft has released a .NET patch that has broken the hack... so this project is dead. Anyway, i wouldn't have used it for demos, as i'm mainly focused now on 4k and 64k intros.

I will try to explain a bit more in some future posts what can be done with those tools for coding 4k-64k intros and what i still need to implement to catch up the 4k-64k challenge.