Finally gotten around to putting VS2010 on this machine, and this time around I’m breaking with tradition and simply not putting MSDN on there, at all. It used to be a no-brainer, put the reference libraries on as you’re going to be looking stuff up all the time. But these days it was always more an exercise in frustration than a useful tool.

Many topics are just “not there”. Huge swathes of really basic stuff are just missing (basic date formatting string specifiers – that’s pretty low level!), so that when you navigate to them it tells you the page is missing. Go online to the MSDN reference there, and you’ll find the page, just not in the locally installed copy. I thought to begin with it was just because I’d installed it badly, but even from a clean install it was still just not there. I’ve since concluded that it must be the Express versions are just subsets of the full documentation, to keep the downloads small. It’s certainly not a functionality split – like they’re only putting in help topics for things in the Express editions – because the Express editions are really quite close to fully functional. No, this is stuff that’s core to .NET and the language.

So, since I’m having to fall back on searching the internet anyway, I figure I might as well have my hard disk space back. The online resources available now are fantastic anyway, and it’s rare that I’m not connected when developing. Most often it’s the online MSDN references that show up first in the search, so in the end it’s much of a muchness – except I don’t have to use the horrible HTML help interface which has been getting steadily worse with every revision of Visual Studio.

I guess this is just another nail in the coffin of the disconnected computer: so many things now expect/assume/require you to be connected to the Internet. Which wouldn’t be so bad, but even with 3G connectivity, a network connection while on the move still isn’t something that can be taken for granted. But I’ll stop grouching about it like an old man, and go with the flow…

Picked up on when reading through some old posts on the sweng_gamedev list, and had to be shared.

I must admit, I’m a bit of a CruiseControl.NET fan. It drives most of our automation systems, and provides us a backbone from which we can hang many different systems. I use it both here at the Company, and when I’m doing tools consulting with other teams. That said, it’s not without it’s flaws and limitations; not least of which is the lack of documentation on custom plug-ins.

There are a wide variety of pre-built plug-ins already available, for source control systems like Perforce and Subversion, and build systems like Nant and MSBuild. These usually work pretty well straight out of the box, if you’re building a vanilla continuous integration server. That is, every change made to source control results in a build of some sort. I’m not going to dwell on why that’s a good idea (it is), I’m only going to say that there are times when you need something different. There are inelegant ways around this, but in truth CC.NET has all of the customisability to let you do this properly within the system – by defining your own plug-ins.

Reading the documentation on CC.NET, you get only a single page of documentation on custom plug-ins, and that page is pretty spotty at best. It shows you a “hello world” plug-in, with no clues as to what information you get into the plug-in, when your plug-in’s methods will be called, or how you’re supposed to pass either status information or logging back to CC.NET itself. In short, it’s pretty useless, other than to highlight that the facility is there, and give you enough of a pointer to get you into the code and poking around.

Sure enough, with Auto-complete on, you find that there are lots and lots of information passed to you, and by debugging and breakpointing inside your plug-in, you can get a feel for when your code is called. Making CC.NET load and unload your plug-in is refreshingly easy and fairly robust, and once you figure out how to pass parameters to your plug-in by specifying them in the config file, you can see the possibilities open up. More importantly than the sample building plug-in though is that, through a bit of digging, you find you can also define your own custom source control plug-ins, and with the combination of those two things, you can do pretty much anything you want.

In one particular situation, I’ve been working with a Perforce (p4) source control system, and a rather black-box build system for the game itself. Rather than wanting a build made of every single commit to the p4 repository, I needed it to make only particular builds – those marked as verified by the developer’s internal test team. This is a pretty common situation when you’ve got an automated build system – you have a raw source control system that lies underneath, which operates at the level of atomic commits. But above that is a logical structure, which only people understand – that operates at the level of ‘nightly builds’ and milestones. You have custom logic which you can apply to the system, using some basic rules. So in essence you have a virtual source control system, built on top of the raw version. By writing a custom source control plug-in for CC.NET, you can expose an interface to your automated build server, so that it recognises when something new is available from that virtual source control, and only builds exactly what you want.

CC.NET offers great flexibility, and those people who develop it, know all about that flexibility. But information on what you can do, and how, is rarer than hens teeth. So over the next few weeks, I’m going to write up and publish here some examples of real-world plug-ins that I’ve written. That should hopefully give readers enough context to go off and write their own plug-ins, to suit their own needs.

Disclaimer: all of the points made here refer to CruiseControl.NET 1.4.x, not the later versions of the system. There are some big and eagerly awaited advancements in the newly released 1.5 version, that many people like myself will have to avoid for now, until it’s bedded in.

Amongst various things I had to sort out today, I was asked to write out a blurb for a potential client about improving build processes and automating/scripting things in the development pipe-line. It’s a subject I get quite passionate about, because unlike so many things in games development, it’s a nice task to do. There are clear, quantifiable goals (“make creating a build a one click process”, “speed up turn-around times for artists by 50%”), and usually plenty of options about how to get there. It is also a nice, self-contained task that you can just wade into and make progress on, unlike for example gameplay coding, where you can often get blocked on feedback from the creative team, having to rework things and so on.

I think that’s possibly why I like to spend time on improving our pipe-line at the weekends or in my off-time; even though I could spend more time on the big pile of client work that needs done, I find myself tackling little bits of our own pipe-line because I know it’s a task I can get done without any other input.

On that note, with some more collaboration with the developers on some little niggles, we finally switched our creaky old makefile based system over to using JamPlus properly. Both build processes still run side-by-side, but the JamPlus version has a fraction of the number of lines in the makefile, runs much faster doing dependency checking etc., and in general is much cleaner and will be more maintainable going forward. I’ll have to walk the guys through what’s there so they can maintain it too, but after that I should be able to scrap the makefiles altogether.

Next step is the art/audio asset to platform binary conversion process, and this is why I really wanted to switch over to JamPlus. Our previous art pipeline would always rebuild platform assets, even if the source assets hadn’t changed. That was fine early on, when all of our tools ran lightning fast and we had few source assets, but very quickly it grinds when you introduce slow tools (such as our font encoding tool that does smart packing of glyphs and colour conversion), or many assets. Also the build scripts which make those assets are all Lua based, and so we have different technology for building code than for building art and audio. I’m pretty hopeful that we can make JamPlus fulfill both functions, and in the process get fast dependency checking for our art assets so that only the assets that have changed get rebuilt. But for that I’ll need a free day, and those are few and far between right now.

There are few things worse as a programmer than to know that you shouldn’t do something, know all the reasons why you shouldn’t, but yet you do it anyway. You tell yourself that it’s more important to get things in quickly, that the code is only preliminary or temporary. You tell yourself that it’s better to get things in so you can see whether or not the whole approach will work, and that taking time on the details now would be premature optimisation.

Of course you know, deep down, that the code that is supposed to be temporary will stay, because it works. Any bad things you put in there will be left with a @todo comment next to them, and shuffled down the priority list. Which might seem like a good thing on the run-up to a deadline, but is so many times more irritating when it comes back to bite you later on.

In this particular case, the standing rule we have at the Company about programming for a fixed memory environment, even on PC, was the one I was ignoring. We were initially developing for PC, and memory handling there can be handed off to the operating system heap management and forgotten about. On every other platform though (especially PSP for which we are now porting things), you don’t have that luxury – every drop of memory has to come from the single small fixed arena, any allocations from outside of that will just fail.

So I had to spend a couple of days, going through all the allocations in the code and plumbing through proper fixed heaps so that it was clear where all the memory was coming from. Two days, and probably that in itself wiped out any gains we made in the original PC development by working fast and loose.

In summary then, no lying to yourself when it comes to weighing up pros and cons of doing something in a way that you know isn’t going to be good enough in the long term. Oh, and work to a fixed memory environment. Even on the PC. It will make your code tighter and faster, give you a clear idea of what your memory footprint is, and help reign in the inevitable problems when your memory usage runs away with you and you have to spend time tracking down who is wasting memory and where.

Virtual memory hides a lot of sins from you as a developer, but it is a crutch which you can all too easily become addicted too. It doesn’t take much to engineer systems to avoid memory fragmentation and waste, but re-engineering them once they are embedded is difficult and problematic.

So I must admit to being a bit of a luddite when it comes to embracing the new languages and technologies available for developers now. Partly this is because I’ve read (and heartily agreed with) No Silver Bullet by Fred Brooks, and partly it is because I’m much more comfortable knowing exactly what is going on ‘under the hood’ when I write software.

That being said, having good tools is a vital part of games development, and to write good tools you have to build good user interfaces on top of functional software. No amount of clean, efficient and well structured code is going to get you past that final hurdle, which is to interact with the user. I have spent too much time on too many different projects faffing around with inadequate UI libraries to want to spend any more on it now. I would say I am comfortable with MFC based development, but I would never claim that it was easy or pleasant.

So when I keep hearing other developers evangelising the merits of tools based on managed code (C#, etc.) and the .NET platform. Apparently it should take the pain out of making tools, and user interfaces, and should let me concentrate on the important things instead. Well, that was enough to tempt me in, and to give it a try.

The thing is though, our engine and game code is all based on C++, simple and clean, and we’re not going to change that (no matter how much XNA and Microsoft try to tempt us otherwise). So any tools we build have to be able to leverage all that pre-written code, and play nice with the other parts of our engine. So we needed a way to make the managed tools work with the un-managed engine, and that was where my headaches began.

Straight out of the gate, building a tool application with Visual Studio 2005 was simple and easy, and it took less than 5 minutes to have a skeleton UI that had all the right hooks for exercising some of the engine functionality needed to pre-process our assets. But then I had to figure out how to link those hooks to our pre-existing code, and that wasn’t nearly so simple. The problem was this – a C++/CLI based application (i.e. our tool UI) needs to jump through a few hoops to talk to native (C++) code. The documentation rabbits on about mixed and pure assemblies, DLL building, COM linking and a whole ream of pages about how to build a managed application. All of which is total overkill for what we needed – a simple wrapper layer between the managed UI application, and the native core code.

Now that I’ve found out how, it’s not as hard as I thought; as I work through the details, I’m going to note them down and post them here, because it was immensely frustrating to continually search for tutorials and references (I gave up on the MSDN documentation), only to find lots of people talking about how simple it was, but no-one bothering to let on how it was done.

Anyway, in lieu of a later, better post, here is what I have so far:

• Make a library DLL, making sure that it has managed (/clr) support enabled. This will form the wrapper layer
• The library DLL can statically link to the native libraries you have.
• Build a wrapper class which uses pointers to your native classes to route commands/requests to the native code. Make sure this is a managed class (it should take the form “public ref class WrapperClass” if you’re using C++/CLI)
• NB: You will have to follow the managed rules about not having native value types in your wrapper classes, but you are allowed to have pointers to native types and use new/delete as normal
• In your fully managed UI application, use the Reference system in the Common Properties for the project to add a reference to the wrapper library. This will automagically allow use of your wrapper layer classes, no need for headers or static linkage. [this is the one annoying step that really wasn't clear from the documentation]

Anyway, a better picture should emerge from this experimentation, and I’m hopeful that once the basic pattern for managed/unmanaged tools emerges, that we’ll be massively more productive and be able to build up a nice tool-set using this new technology.