Gah. New rule for Subversion repository maintenance – run “svnadmin verify” as often as you run your off-site backup process, and arguably don’t do the backup if it fails. One of our support repositories (as opposed to a development repository which I’m a bit more paranoid about) has had a dodgy revision in for a few months now which would have bitten us had we had to restore from scratch. It looks like it was a failure while checking in a massive binary file – it doesn’t affect the day to day running of the repository, but it means that we can’t dump or load (and correspondingly can’t effectively restore from backups).

Since rebuilding the repository revision by revision is always a massive pain, I’ve done some mucking around in the guts of the repository to get around the problem. And since my Googling of the issue has been less than helpful, I thought I’d post here to give a reference for anyone else with a similar issue.

Symptom:

Running svnadmin verify on the repository results in a “Checksum mismatch while reading representation”. The output here is misleading, because it will say something like “* Verified revision 23″ on the line before the error message. This means that it is in fact revision 24 which is bad. You will also find that if you try to dump the repository, it will successfully dump revisions 0 through 23, but then fail on 24. If you try to dump revisions 0:23 and then 25:HEAD like I did, you’ll probably find that the 25:HEAD revision doesn’t work.

Diagnosis:

One (or more) of the changes to files in the revision that is causing problems has a different checksum than the one that the revision file recorded at the time. So when svnadmin verify looks over the contents of the revision and recalculates the checksum it finds that they don’t match and tells you. This means one of two things: 1) the checksum recorded at the time was wrong, and the data in the revision/file is valid, or 2) the data in the revision/file is corrupt, and the checksum at the time was correct.

If the file generating the bad checksum is a text file, you might be able to look at the contents of the revision file and check if it’s noticeably corrupt. If the file is binary as mine was, that’s probably not an option. Even more so if the file is large (mine was several hundred MB).

2) seems to me more likely, so chances are the file in question is corrupt and you need to fix the data. But if 1) is the case, then all you need to do is fix the checksum. Either way you probably can’t tell at this point – so best to assume it’s gone and work from there, or at least treat it as suspicious and verify it against other sources for the data if possible.

Possible Solution:

If you’re happy to assume that file is corrupt, then you can get your repo back to a verifiable step by changing the checksum saved in the revision file to match the checksum which will be generated from the data as it is now. The data won’t change so you’ll still have to verify it manually or delete it later, but at least you can persuade the repository that you don’t care.

Process:

I’m assuming here you’re working directly with the server on Linux. I use Debian, so tools like grep and hexedit are usually available (although I had to install hexedit).  The same principles would apply on Windows, but the tools would have to change.

1) Identify the revision which is corrupt. This is straightforward – it’s the revision after the last successfully verified revision

2) Identify the file in the revision which has the bad checksum, and find the bad checksum in the revision. This is harder – the revision files (stored in /repository/db/revs) are binary, and in my case, huge. But grep is your friend here. svnadmin verify gives you the checksum that is currently recorded – this is stored in the revision file, right next to a description of the file. Here’s a grep command that searches the particular revision file for the checksum we’ve been given:

grep -e "79a1686d0dfb8618b8ccfc9eb7d74759" -A 3 -B 3 -b -a main/db/revs/24

384989609-id: 5cu.0.r24/384989609
384989633-type: file
384989644-count: 0
384989653:text: 24 75689685 293851064 294285337 79a1686d0dfb8618b8ccfc9eb7d74759
384989724-props: 24 384989543 53 0 113136892f2137aa0116093a524ade0b
384989782-cpath: /path/to/the/bad/file.exe
384989842-copyroot: 0 /

The number at the start of each line is the offset, we’ll use that soon. The cpath line is most interesting – this is the file you can expect to be corrupt. But it’s the :text: line that we need to change to get things working. As described here, (look for the section on the revision file format) this line is of the form “<rev> <offset> <length> <size> <digest>”. We don’t want to change the first 4 parameters – they’re most likely just fine. But the 5th parameter is the bad checksum, and we’ll need that in the next step.

3) Change the bad checksum to match the “actual” checksum which the svnadmin verify process is coming up with. Again, this is printed out when you run the verify. To make the change, I used hexedit, which thankfully doesn’t try to load the entire (huge) revision file into memory. You just fire it up, and press Return to enter the offset within the file to jump to. It wants it in hex, so a quick conversion turns 384989653 into 16F279D5. From there you can press Tab to switch to ASCII editing, quickly find the offending checksum and overwrite it with the new, valid checksum; then press Ctrl-X to save out the file and exit.

4) Re-run svnadmin verify. It should now successfully verify the broken revision and move on. If it doesn’t, check to see if the revision and checksum it’s failing on are the same – if they’re not then you have more broken files/revisions, and you should repeat steps 1 to 3 until they’re all gone. Hopefully there won’t be too many of them. And remember – just because your repository is now verifiable, doesn’t mean that your data is valid. All you’ve done is told the svnadmin tool that the checksum for the data you have is the same as the checksum it expects.

Here’s a little spiel I wrote up in response to a student doing research who was asking why we set up in Edinburgh rather than Dundee, and what I thought of Dundee as a creative hub. It’s got a bit of history of us in there, so I thought it would make a good blog post.

For us, really, it was a convenience thing. I studied in Edinburgh, and my first job (VIS Entertainment) was based in Dunfermline, so I commuted there for a while before they moved their office to Edinburgh. For 6 months or so I was seconded to the Dundee studio for a project, and the 2 hour commute each way took a real toll on my quality of life. Compared to that having an office in Edinburgh was a breeze, certainly much more social.

I’ve always loved Edinburgh as a city, it is an amazing city, with the warmth and personality of a smaller town, combined with the incredible history and range of night-life and culture that I think comes from it being a capital. So the best of both worlds I always thought. Glasgow is larger, but for me lacks charm and friendliness; Dundee has that charm and friendliness but lacks the same range of things to do that Edinburgh or Glasgow offer.

So all business motivation aside, I’ve always much preferred Edinburgh to the other options. When VIS went out of business in 2005, I had just moved flats and was committed to at least another 5 months of the lease. With that in mind, it was either Rockstar North, commuting to Dundee for a position with one of the various teams there, or trying to start my own studio as I had been thinking about for a while. With my own quality of life firmly in mind, I chose the latter, and started the studio, running it out of our spare room. The first 9 months or so was a struggle to find work, but I realised that there was a niche here in Edinburgh, for those developers who didn’t want to disappear into the behemoth that is R* North, but who (for personal reasons like me) couldn’t face shifting to Dundee, either relocating or commuting.

So rather than follow the herd and shift up to Dundee, I was determined to take advantage of both the personal enjoyment of living and working in Edinburgh and this talent pool. I figured that I would have a better chance recruiting people if I was running in Edinburgh, both experienced developers who wanted out of R* or Outerlight for whatever reason, and other people who would like to come and live in Edinburgh but who might not want to live in Dundee.

Of course, most of our business involves dealing with other games developers, and we do have a good working relationship with several studios up in Dundee. I do find that I spend a non-trivial portion of my time on the train up to Dundee to meet with those people, and that would be much easier if we were based up there with the other small studios. But many of our clients are down south as well, and so Edinburgh is well placed as a travel hub to get to all of those destinations.

Aside from the potential of easier collaboration with the many studios who are Dundee based, the other important consideration is premises rent. I gather that many of the studios who are tempted to the Seabraes developments are given massive discounts on the rent there, due to the council/Scottish Enterprise’s efforts to create a digital media hub. That’s a great thing, and certainly must weigh heavily on any manager who has to find space for their team. But for us, especially since we started out working remotely from home, this was never a pressing issue. When we reached a sufficient size where a shared team space was a good idea, we shifted out into our Palmerston Place office. By that point, with two developers, the proportion of our monthly outgoings which were rent was very small; with four developers it is even smaller. Salary has always dominated our cash-flow, and so rent price is less important than quality of life for the team.

We’ve been especially lucky in that both of our offices have come in at around 100 UKP per person per month, which even for Edinburgh is very cheap. Largely that is down to good fortune and timing, but it’s also because we’ve been prepared to look at non-traditional office spaces. Our first place was a room in a Georgian town-house in the West End of Edinburgh. The décor was fairly shabby, the services provided were minimal, but the place was nice; it had real character, and it was in a central location so that everyone on the team could get to it easily. There are plenty of pubs and shops and bus stops nearby, which makes it much nicer to work because you can get life stuff done. The new premises are set to be much nicer, the space is larger and it has a nice comfortable feel to it. The place is still being done up (hence the bargain rent), so it has many rough edges, but it feels like a good creative space. And again they are central so that quality of life is high.

The key thing I think for us is that the common feeling about Edinburgh (that property prices and rent are high) doesn’t hold up under scrutiny. We had quite a few options for office space, all within easily affordable price range, all central Edinburgh, all spaces in old buildings. The impression I get is that the myriad of old buildings in central Edinburgh have many quirky, odd spaces, which don’t allow the big, open-plan offices that companies seem to like these days. As a result, once your business expands beyond a certain size (10 or so people), it is very difficult to find a single space to suit, and so businesses have to go out of town, to spaces like South Gyle and Leith, before they find modern office-space that fits their specifications. And so the prices out of town rise, and there is a glut of small-sized offices slap bang in the centre of Edinburgh which keeps their prices reasonable.

As a small, creative business, we can make pretty much any office space work. We don’t have to have an open plan space, we would have been quite happy taking two mid-sized rooms in a town-house next to each other, and just wandering between them. So by being more flexible about the types of space we rent, we can fit into any of the odd shaped offices which central Edinburgh offers.

I realise of course that your questions are all relating to Dundee, and I’ve only talked about Edinburgh, but the answer to me is clear. We are based in Edinburgh, and unlikely to move, because:

1. Edinburgh’s great
2. Edinburgh’s cheap (enough for us)
3. Edinburgh has a talent pool we can take advantage of that a Dundee company can’t
4. Shifting to Dundee would make it easier to work with other Dundee businesses, but harder to work with everyone else

There are many good people and businesses that work in Dundee, and they probably disagree with me on the personal preference for Edinburgh as a city (in fact I can think of several off-hand who do, and tell me so on a regular basis); they would be better people to ask about why Dundee has been such a success. As an outsider, I certainly feel that incentives have played a part, but I think that probably a bigger factor is that the studios in Dundee are often started by Dundonians. And there are some very talented Dundonians, and the very fact that there are a lot of talented teams there already encourage more teams to set up alongside. So Dundee has a critical mass of talent, which itself helps to keep other talent nearby (its nice to know that if you’re relocating to somewhere like Dundee, that even if your new job goes belly-up, there’s plenty of other jobs nearby so you don’t have to relocate again).

The telling fact is I think that when one of those studios dies (e.g. Visual Science), the people involved with that will start up new businesses. The fact that those founders choose to start up again in Dundee and not relocate somewhere else does suggest that Dundee is a good place to run a creative team. Whether the incentives are forming an unnatural situation (i.e. those businesses would have moved elsewhere had incentives not been offered) or not is an interesting question, but one better answered by someone who’s started a business and chosen Dundee.

And we’re back! By which I assume that the scheduled down-time while our server was migrated to another data centre has been completed without a hitch. I did intend to post here that it was going to happen, but it did in fact catch me by surprise because I thought it was supposed to be last Friday.

My only real mistake was to start reading some of the comments on the Dreamhost status blog regarding the move. Despite the fact that they were open and up front about the move – giving warning on the status page as well as by email more than 2 weeks in advance, they are still customers who feel free to bitch and moan in a most personal and horrific way. Seriously – this sort of stuff really makes me sad to see the sort of attitudes people have these days. Nowhere in the Dreamhost sales pitch does it imply that your 8 USD a month was somehow buying you a service that would be up for absolutely 100% of the time, and yet these folk think that even the best handled server downtime is grounds for some personal abuse. If I were running Dreamhost, I wouldn’t hesitate to respond to any of those “this is unacceptable” comments with “Oh I’m sorry, here’s a refund for the rest of your service, don’t let the door hit you in the arse on the way out”.

Anyway, trust in Dreamhost’s tech staff aside, my paranoia has been in high overdrive recently, so I had taken this opportunity to double check that our nightly backup procedure was working properly – and I was pleased to find it had. Which means that, should Dreamhost fall over at a critical juncture, we can fall back on our local mirror of the server and only lose work done since the previous night. Not that I’m expecting DH to fall over – they seem to have gotten on top of their random downtime issues, and since I’ve been tracking it with an external tool we’ve only been off-line for 4 hours out of the last 6 months, and no more than 2 hours in a single outage. Why don’t we just host our work locally and use an off-site backup? Well, we do a lot of collaboration with other client, and upstream bandwidth from the office is rather precious. That and the fact that Dreamhost already have quite a few mechanisms in place to restore connection and hardware problems, so they’re probably far more reliable than our local server machines anyway.

Of course, even that’s not enough for us, paranoia wise – not only do we have the primary copies in a well maintained place off site, and secondary copies in the office, we also keep physical (DVD) copies of the repositories in yet another location. Of course the first two are all automated, but the third requires me to actually go over and poke the office server to make a DVD and take it away with me. Unfortunately my attempts to train the local squirrels to do automate the process for me have been unsuccessful, but I have high hopes…

Okay, so we have had our Jobs page up for a while now, but ever since we started I’ve had a continual stream of emailed applications from potential employees. Some are impressive, some are not, but this post isn’t directed at any of them in particular. It’s intended as a guide to anyone else who is thinking of applying, so I’ll put a link to it on the right hand side.

First off, one of the biggest positive factors for a prospective employee is to show enthusiasm, both for developing games and for our company. We have this web-site and blog – it’s always been there, it’s fairly open. You could readily skim over everything on it in 5 minutes. So if your application clearly shows that you haven’t put in that small amount of effort, expect it to be either binned outright, or at least given short shrift. I make an effort to write a personal reply to every application, but the ones that annoy me get very little attention.

That point is at the core of why most applications that turn up in my in-box will fail, but I can sum the rest up in bullet points (most of which represent actual applications).

1. DO understand that you’re applying for a games job, not just a ‘software’ or ‘IT’ job. Those who end up at games companies are primarily those who have a real passion for games, and that comes through readily in a good application.
2. DON’T show ignorance of who we are and what we do. There are two of us, and we primarily do custom software development for other games studios. And yet we get applications which:
   1. talk about things like meshing with our organisation and looking forward to working in a large team.
   2. ask about jobs on our QA team
   3. are an impersonal email, even though the applicant worked with Pete and/or I at VIS
3. DON’T send a generic “please I would like a job at your company” email. This includes:
   1. not mentioning our company at all
   2. not making sure the cover letter is appropriate to our company,
   3. the worst sin of all: sending the exact same email to every games developer in Scotland, and leaving all of those email addresses clearly visible in the To: line of the email!
4. DO make sure your email and CV are free of spelling and grammar mistakes. Bad spelling and grammar says to me: “I’m so slack that I can’t be bothered to spend the 15 minutes it would take to look them over properly, despite the fact that I’m applying for a job that might be hugely important to the next few years of my life”. Frankly that’s not a person I’m necessarily keen to employ.

Paying attention to these details take a very short amount of time. There aren’t so many games companies out there that you can’t do them for every place you apply to. Personally I think for the potential reward (i.e. getting a job), they’re worth it. I’m sure more bad examples will appear in the future, so I’ll be keeping this post up to date.

It occurred to me while watching a re-run of a pilot for a particular popular sci-fi show, that perhaps people worry too much about the plot for their sci-fi games. Sure, the Phl’aarg forces might have strong and compelling reasons for attacking the Kerflump home-world, but let’s be honest, your average game player really doesn’t care.

In my view, the job of a back story for a game is to allow willing suspension of disbelief for long enough that the player can enjoy the game itself. If the game is fun, the player will remember the story with fondness, even if it does follow closely the plot of some more famous film or book. The key factor is that the story shouldn’t make the player cringe, and shouldn’t contain the kind of horrible glaring inconsistencies that they can’t ignore. Some of the most effective game back stories are the simplest. Mario has to rescue Princess Peach. Sonic has to free little animals from the evil Dr. Robotnik.

If you are going for a more complex story, consistency is more important than fleshing out details. Once you’ve persuaded the reader/player to make the first jump to your story, try not to force them to make any more. You might persuade them to accept that magic is real, or that technology allows you to move instantaneously between points, but don’t change the rules afterwards to suit some awkward bit of plot development. Choose your universe rules at the start, keep them consistent and simple, and then base your story around those rules. The player then has a nice consistent world view, they understand what’s going on, and they’re happy because there aren’t constant surprises.

If you omit details, the player will happily imagine the rest of the universe according to the basic world rules that you’ve established. Take Star Wars for example – you are shown a very small section of the universe, and the rest is simply implied. By not fleshing out endless irrelevant details, you avoid accidentally introducing glaring inconsistencies, and also avoid boring the player. Take it from the point of view of a character in your world. They don’t care about the mechanics of inter-stellar travel, they’ve been living in a world with it for a long time. It is mundane to them – they wouldn’t dwell on the details, they dwell on how it affects them. So don’t follow the Star Trek model of baffling the viewer with science details, instead show them the parts which affect them – e.g. travelling between point X and Y will involve hyperspace, and will take Z amount of time.

Well, at the risk of sounding like a total Nintendo fanboy: I’m delighted with the Wii! By virtue of some late night queuing at a 24hr Tesco by Pete on Thursday, we picked up a couple of consoles, and Zelda/Rayman. I’ll leave Pete to give his impressions of Zelda, I’ve been looking at Rayman and Wii Sports.

Basically it’s been a great experience right from the start – install went smoothly; all the parts feel solid and well constructed. The Wii-mote and nunchuck have a comforting weight to them, and don’t feel flimsy. I’ve been using the wrist-strap out of paranoia, but haven’t lost control of it yet. Last night was a wine fuelled extended session of Rayman with the girlfriend, who seems just as pleased and hooked by it as I am. I was watching her reactions to it carefully, as she’s really not much of a gamer; however she picked up the controls without any trouble at all, everything seemed natural and intuitive out of the box.

Rayman itself is a little esoteric, but the games make me laugh, and the lovely characterisation of the mad rabbits is fantastic. While I think it has some replay value, I think Wii Sports will probably get played more long term – especially with the fitness training thing. A lot of the Rayman mini-games and Wii Sports  games involve energetic movement, we were having to take it in turns to play as after a few we were out of breath!

Only minor gripe so far has been the fact that the News and Weather channels are offline, and that the Wii Shop channel failed to connect first time round. Other than that the experience has been great, and I’ll certainly be buying at least another Wii-mote (probably with the Play game) for multi-player fun. The girlfriend will probably be taking it to her family over Christmas as well for some fun games too.

Anyway, fan-boy ranting aside, I predict a glowing future for the Wii, and development on it. While the unique control mechanism is interesting and well implemented, I don’t see it standing in the way of traditional console gaming – since you can get just as much control from the wii-mote/nunchuck pairing even without using the motion sensitivity of the controls. I’d love to have the opportunity for us to develop on the platform, but we shall have to see what pans out.

…has been made, based on this breaking news. I was humming and hawwing about whether I even wanted a console, or whether I’d stick with PC (all the recent titles I’ve wanted have been on PC, bar none).

Speaking of which – stand-alone Tremulous has finally been released! No more need to dig out the dusty old copy of Quake III you had.

Boy do I need a holiday though – its been 5 months straight now without a day off (or at least, not one that I didn’t have to make up at the weekend). I’ve been trying to put aside time for side projects (notably a game pitch demo using Torque and some more website work), but its all been taking second place to the main contract for Barco and some other part-time remote work. Not to mention all the paperwork and faff that goes along with the financial year end at the start of this month.

Still, coming up to the end of the Barco work at the end of this month – high on the priorities list are: getting VAT registered (so I can splash out on a shiney new lap-top), finishing up this game pitch demo, catching up on all the GDC/E3 footage, and a hundred other little things on the to-do list that I can’t even think about until I can get a good solid week-day in the office. But all of that can wait until I’ve taken a few days to relax, as I think I deserve a break…

To be found here at Lost Garden. Worth reading, if only for the comedic diagrams near the start.

Amen

News in today that Peter Jackson’s production company will be working with Bungie, Universal and Fox. Good news I think – it certainly seems a better candidate for a game-to-movie conversion than Doom. Plus the Halo ring structure is kind of outdoors (discounting the unusual sky backdrop), so they get to set it in New Zealand and pump some more money into their economy. I always like to see the Kiwis get ahead (although I’m compelled to admit its because I’m secretly hoping they’ll all get so rich they’ll start giving away free plane trips over there).

Of course, game-to-movie conversions are always going to be at least a little cringe-worthy, even when the game story is good, the movie can’t fit everything in (like a book-to-movie conversion) and it loses its magic. At least movie-to-game conversions can expand on the movie’s plot and get you more involved in the world.

One good example I found recently though was the book that Raymond E. Feist wrote after the success of Betrayal at Krondor. I’ve never played the game (I know, I know, how can I call myself a gamer), but I can see the shape of the game in the book, and feel just how the various stages would map to quests and sub-quests; and how the character/group mechanics would work. You can even almost see the levelling mechanics at work in the character of Owyn. If only all medium crossovers were so effectively possible.