For too long now, I’ve been attacking my to-do list in a rather sporadic way. RememberTheMilk.com is a wonderfully flexible way of storing and categorising a task list, and given how mobile I am the web-access is great. The idea that on my own I would be able to remember all of the myriad of different things that I need to do on a regular basis to keep the business operating smoothly has long since been discarded as a pipe dream. Tax returns, bills, server maintenance, paperwork, and that’s not even counting the one off tasks which I can’t do right away but can’t afford to forget.

My problem, as is pretty typical for a task tracking system, is that it needs to be a matter of habit to check my task list every day. Which I do. But the very act of having a task list is by definition a triage action for managing things I need to do, and so if I’m not tackling the tasks quickly enough then they start to accumulate. If my list of due tasks is clear, then it’s fine, because I then look ahead, see what’s coming and act on it in advance. But every task has it’s own priority level – 1 means ‘can’t afford to let it slip’, 2 is ‘must do, but the world won’t end if it is delayed a little’, and 3 is ‘needs done when you get a chance’. So when time is pressing, the priority 3 items get left, and so my overdue tasks list is no longer clean and empty, but now has an item lurking there, untouched.

Once that psychologically important barrier is breached, then it’s oh so very much easier to let the next item on the overdue list slip as well. After all, I know I’ve fallen behind, but it’s okay because they’re all low priority. Before too long, the overdue list has a dozen items, and I’m no longer tackling tasks in advance, I’m just picking off the important ones when they appear on the overdue list. Some of the items on there are 2 months overdue, but even though they’re low priority, I don’t want to just change their due date to the future. That would be hiding from the problem – they are two months past due, and I should really tackle them.

And so, since I’m down here in Reading this weekend, and I’ve already done most of a day on our Evolution work, I’ve taken some time and made a concerted effort to tackle every last item on the list, even the low priority ones. Including the last one – “write blog entry”.

Done.

You know, I wouldn’t ordinarily mock applicants to the Company, but sometimes I get someone who makes my teeth grind so badly that I can’t help it. Such as the email I received earlier this week, entitled “Application for the post of Web trends Technology Ananlyst”[sic]. Hmm. Yes. Right. Spelling and capitalisation issues aside, WebTrends? What possible reason could you think that we would have to use WebTrends?

To be fair, this girl's been hard done by, as this image is used everywhere despite being a blatant photoshop job. But to indicate idiocy in all its forms, you can't beat it.

I refer potential applicants again to this post, although it should be noted that the unspoken rider to that post is that morons need not apply.

Right, I’ve had enough of the sad kitten at the top of the blog now; you can only stand so much cuteness before the mind rebels. Time for a quick update on status, as I’ve been quite heavy with the waffling and opinionated posts recently.

This post is written courtesy of the wi-fi in a B&B in central Reading, where I’ve been installed for the last week. It’s not my favourite accommodation in the world, I’ll admit (far too many chavs hovering outside the window when the pubs kick out). Luckily, I’ve found a decent room to rent which I’ll be taking up from the end of next week, where I’ll be for the majority of the next few months. Why? Our new client of course – the behemoth that is Microsoft Games.

This is indeed the tools gig I hinted about previously, but sadly that’s as far as I can go in terms of details; not because I’m working on anything super-secret, but because the contract is just generally confidential. Suffice to say it will allow me to indulge my passion for making process improvement tools and automated build systems, and deploy them on a scale that is far beyond our range as a tiny software shop.

Sadly though this requires me to be away from Edinburgh for much of the time, which I’m still getting used to. I’ve lived in Edinburgh for so long it’s hard to adjust to living elsewhere, it’s a city that spoils you for anywhere else. This will also leave Tim minding the office in Edinburgh on his own, but hopefully he won’t be rattling around the place too badly. I’ll still be working with him remotely on our work with Evolution, but he’ll have to keep Bertie alive on his own…

On the bright side, since I’m kept away from all of my usual distractions in the evening, I’m hoping to use the time productively to get some serious effort into our internal prototypes. That being said, in the 3 weeks away so far, I’ve managed no more than a few hours, but I put that down to the fabulous selection of pubs in and around the Rare studio in Warwickshire where I spent the first fortnight – it’s really hard to feel creative and productive when you’ve just had an exceptionally tasty portion of steak and chips for dinner! I’ve settled for keeping the usual pile of paperwork under control. Speaking of which – I must sort out last quarter’s VAT return before I miss the deadline…

 

Why don't you have a decent search facility WiiWare? Why?

It’s true. One of my biggest issues with the games industry as it stands today is with the digital distribution stores (DDS for brevity) in place on the various platforms. I’m not going to jump on the bandwagon with others who have predicted the imminent death of physical retail stores; I think there’s still a large place for brick-and-mortar game shops, and they’re certainly not going away any time soon. But I think a large part of the continuing need for retailers is down to the failings of the various digital providers. Let’s list the most relevant ones:

• Amazon
• Steam
• WiiWare
• XBox Live Arcade
• Playstation Network
• iPhone App Store

Amazon of course isn’t really a DDS, although I believe they’re changing that. It’s really just a retailer of boxed products – the shop-front might be on-line, but the products are generally posted to you; however the problems it faces and has overcome are very much relevant to all of these services. Steam is much more relevant to the discussion here, as it’s a proper DDS, and it has learned from many of Amazon’s lessons; sadly it is let down by uncompetitive pricing and the lack of community integration.

Really though, my irritation comes from the remaining 4 DDS – each of which is the only means of buying product for their respective closed platforms (Wii, X360, PS3, iPhone). All 4 suffer from the same problems, all of which have known solutions as demonstrated by Amazon, Steam and others. And the 4, together or separately, represent a massive market of game-hungry users, with cash to spare, who just want to find the good games and ignore the crap.

Here are the main problems, in order of importance to me (the user):

1. Navigation: How do I find games that I want to buy
2. Selection: How do I choose when I’ve found those games
3. Purchasing: How hard is it for me to buy the games once I’ve chosen them

Navigation is the real fundamental problem here. All 4 providers suffer from the same issue: their services are popular, so developers make many titles; users are then swamped with choices. Without any external information (reviews, friends’ recommentations), all products look mostly identical, with only a superficial information (title, image, etc.) to distinguish them – assuming the user wants to read through every title’s description in the hope of finding something they like. If the average quality of titles is low (i.e. shovelware), then great titles are lost in the noise of rubbish, and customers are forced to take a punt on titles when they have little idea of their quality. Once they get burned once, they’re reticent to come back, and likely to dismiss the entire shop as shovelware.

All 4 holders recognise this as a problem, but take varying strategies to get around the issue:

• Top X lists (sales based): Popular products are easy to find. Great. New products have little chance to generate sales because the titles in the top X list keep selling (because they’re the only ones the user can readily find).
• Title searching: Allow the user to search for a keyword in the title or description. Great. As long as the user knows what product they want in advance. Little to no chance of discovering relevant products.
• Limit the number of titles in the system: The console DDS do this more than the iPhone, simply by maintaining high barriers to entry (requiring approval prior to development, enforced QA standards, etc.). But at best this delays the problem from becoming serious. XBLA recently wanted to implement a policy of culling poorly reviewed/low selling titles which was a clear attempt to tackle this issue, they’ve since backtracked on this in favour of better searching (yay!)
• Highlight particular titles: XBLA prefers this approach – titles get a week of being featured prominently on the front page. Great. Now you have to make enough sales during that crucial week to build enough momentum to get onto the top X list. Miss your week, and you’re shafted. Better hope you’re not featured during the same week that GTA4 comes out, eh?

The approach of limiting the amount of titles in the system is pure short-termist madness. Maybe it is just a short-term fix until a proper storefront system can be made, but XBLA has had what, 3 years now to mature their navigation systems? The solution is one already demonstrated by Amazon. Navigation is the key issue. Searching is only one potential fix. Products need to be categorised into groups so that users can find the set of products they like by interest. Products need to cross link to each other: “Liked this title? Why not try X and Y, also from this developer?” “Customers who looked at product X ended up buying product Y and Z.” “Customers who viewed these titles,” etc.

Random title prominence: this is so underrated. Sure, the front of your store is prime real-estate, and you probably want to sell it, but you can come up with a system which allows games to be featured if they’ve paid, or if the users have rated it worthy.

I can see the DDS people’s defence: “that’s too complicated a UI to put on a console, it needs to be kept simple”. Well maybe you’re right. That brings us straight to point 3: ease of purchase. Why is the game store only on the console (or phone)? It needs to have a properly integrated equivalent on the web. Customers like shopping on the web. They prefer it. They’re used to it, it’s more flexible, and it supports a much more pleasant experience. Ever tried to enter your credit card number using a joy-pad? It’s not fun. Why are you making me do it? I want to be able to browse a game-store on my PC that gives me as much functionality as Amazon, purchase my game, and then press two buttons (Shop, Download Purchased Titles) on my console to get that game downloaded.

Sure, some times it’s nice to be able to buy direct from the console, but it’s not my first choice. Keep it there as a more limited option and I’d be fine with that, as long as the web-store was nice. But as a developer, I want to be able to publish links to my game on a web-store, so they can get straight to our games, and get them onto their console in minutes.

Back to point 2 though – choosing products. I don’t trust reviewers as to what games are good. I certainly don’t trust the platform holders, since they have a financial interest in the products doing well. I trust the customers. Not individuals, because there are clearly nut-cases out there that rate highly or lowly depending on whether they took their medication this morning, but aggregate ratings over time.

Tell me what games sold big in the last week, or month (doesn’t have to include numbers). Tell me the average rating in the last week or month, and how many people rated it. Publish customer reviews, and professional reviews, and metacritic scores. Put all of the rating functionality into the search system, so you can find titles that rated over 4 stars in the last month in the flight simulator genre. Show me the all-time classic RPGs, based on ratings since the store first open. Maybe I’ve a hankering for high quality old-style adventure games, let me find those.

None of this is crazy blue sky thinking. It’s all been done, it’s all been shown to have worked. Build a better DDS, and you’ll sell more products, we’ll sell more games, the customer gets more games, and they get better games so they come back and buy more. I can’t think of any good reason why they wouldn’t want to fix their stores, other than to make little kittens cry.

(…continued from part 1)

First off, this implication that number of hours you work has absolutely zero relation to the level of talent you possess. I’ve worked with talentless hacks that “worked” 60-80 hour weeks and still achieved less than everyone else. I’ve worked with amazing people who worked bang on 9-5 Monday to Friday and were some of the most able and committed developers. Of course there are extreme cases – talented folks who love their job so much and are willing and eager to work extra hours, just as there are people who are so apathetic about their work that they don’t even want to do their contracted hours, and coast at every opportunity. Talent and the number of hours worked are independent variables. To try to connect them is not only foolish, it’s selling your employees short.

The coasters need to be sacked, plain and simple. Those who work extra hours but aren’t very good should be given the chance and help to improve, but if they can’t they need to go too. Those who work the 40 hour week and produce are the core of your business, and need to be treated like the stars they are: not only are they good at producing, they are sensible enough to do it in a sustainable way. Those who are great and want to work all hours can be gems, but only if they are managed properly. Try to keep them to a sensible working week, and they’ll stay gems for longer; if you don’t they’ll last a while, and then burn out.

Burning out

We’ve all seen it, many times. Constrain their hours and they’ll find other ways to excel; they’ll cram the same amount of work into the time they have, or they’ll go away and enjoy their evenings, eager to come back in the morning, brimming over with good ideas that they’ve been thinking up while they were away from work. But most importantly they’ll have had a life outside of the work, and that will make them happier with their lives and happier with their jobs. If you let them work those crazy hours, you are both taking advantage of their generosity, and setting an unreasonable precedent that other individuals on the team should be doing the same. No matter how much you tell your employees that 9 to 5 is fine, they’ll look at the long hours those few are putting in, and feel that by not doing those hours they aren’t pulling their weight.

Most of all I detest the idea that making games is special, and that somehow by getting to make games we forfeit some of our rights because we get to do work that we enjoy. Screw that. I make games for a living because I love it. I’ve already sacrificed the higher salary I could get in the regular software world. I’ve sacrificed the stability you get outside the games industry. Now you’re trying to tell me I should sacrifice my quality of life as well? And to top it all off, I should be thankful to those who pay my salary for the priviledge of doing my job? No thank you.

When I’m making games, I’m doing a job. I deserve to be paid for the job I do. You want me to sacrifice quality of life, you pay me for the priviledge. My love for what I do comes out in the quality of the products I make, and that’s the only outlet there should be for it. Making games needs passion because the games themselves need love to make them good. By asking for ridiculous working hours or low wages, you are asking me to be less passionate about my job because to do it I need to accept being screwed on pay and conditions.

I’m not some idealistic student, I know that it’s not as simple as just clicking your fingers and suddenly we’re all working standard hours for good wages. The business needs are always going to come first. Sometimes the deadline will loom, or things will go wrong, and we’ll have to work long hours to put it right. We are dedicated people, and exceptional circumstances require exceptional measures. But right now, the individuals working in games development are being regularly asked to subsidise their employer’s costs by means of their own time and effort. Worse than that, many of those employers think this is fine and right and just the way things are. Wrong. The sooner we accept that abusing our staff is unprofitable in the long term the better off we will be as businesses. The sooner we accept that the 40 hour working week is the norm, and everything we do should be trying to get us closer to that norm, the better off we will be.

And so the perennial topic of working hours comes back to us again, this time as a result of some spirited discussion from the IGDA. The exact nature of the discussion has been covered well elsewhere, but suffice to say that an IGDA board member (Mike Capps of Epic) has been lambasted by the game developer community in general for his statement that Epic doesn’t want to hire the sort of people who just work 40-hour weeks.

 

Slaves to the Grind

There are two parts to this issue really, the first fairly obviously is the problems that this has raised in the IGDA itself. Rather than take what seems to me the obvious route (make a public statement that Epic, while being free to run their studio any way they see fit, is choosing to operate in a way at odds with the IGDA’s stand on quality of life for developers), when pushed on the matter, the rest of the IGDA board has essentially folded, almost entirely on their QoL position.

Now for an organisation that has put QoL very high up on its list of priorities, this is a real problem. The developer community (or at least, that part of it that I hear from and talk to) has always struggled to see the value in an IGDA membership; it doesn’t provide much in the way of tangible benefits, and the social and networking aspect varies massively based on the activity of the local chapter in your area. Certainly the IGDA doesn’t provide anything like the benefits that a union would, as it’s specifically written into its constitution that it cannot become a union or anything like one. But the IGDA’s advocation on quality of life issues has always been one of the big pluses for it in my book – it occupied a niche in that it is placed to represent the best interests of its member developers in campaigning for a better industry for us all. To take away that advocacy position seems to me removes the biggest reason to support or recommend membership to others.

The furore surrounding both the original statement by Mike Capps, and the subsequent IGDA refusal to condemn his stance on general principle is what confounds me though. There is no reason why individual members of the IGDA, even board members, have to work for studios that slavishly follow ever policy that the IGDA might recommend. And with that in mind, it’s perfectly acceptable for Mike Capps to remain a board member, even when his employer’s position conflicts with the recommendations of the IGDA. The next time the elections come around, the IGDA membership will think long and hard about whether or not it’s a good thing to have a board member whose personal policies conflict with such a high profile policy of the organisation. Great. That’s democracy in action.

No, what bothers me is that the other IGDA board members have steadfastly reversed their own organisations positions rather than criticise another board member. They even seem to have gone so far as to implicitely defend Epic’s policy. What, exactly, is the point of saying “all studios should aim for a 40 hour week, because it’s better for everyone involved”, if you then follow it up by saying “oh, unless you’re an IGDA member already; in that case you can run your studio however you like, and I’m sure Epic have a good reason for preferring a longer work week, they do seem to be quite successful and all.” The whole discussion on the IGDA forums has been flabbergastingly forthright in its defence of the over-working of individuals in games development.

The board members, and other senior figures in the IGDA, all seem to be quite surprised at the vehemence with which they’re being attacked. They don’t seem to see their own stance as hypocritical, but attempts to get them to justify their position have resulted in only anger and latterly heavy-handed moderation to quell the continuing argument on their own forums. More recently they have come out with public statements to clarify their position, and to attempt to re-assert their original position on QoL issues, but it all smacks of too-little-too-late unfortunately. The board’s own defence (both implicit and explicit) of Epic’s practices have in my opinion exposed their stance on QoL as nothing more than lip-service towards the ideal, despite the obvious importance the issue has with their (non-management) membership. If that membership hasn’t already voted with their feet and left by the next elections, I hope they show their dissapproval and vote out the incumbent board members in favour of some who actually believe in the policies the IGDA publicly endorse.

All that said, I’m not an IGDA member, and with all of this, I have no intention of becoming one now. No, what worries more is the attitude shown originally by Mike Capps, and latterly people on the IGDA forums. That somehow a person who only wants to work a 40-hour week is just a jobsworth, there marking time and collecting a pay-cheque but with no real passion or involvement with the work they do. That somehow the number of hours you work is linked to your talent. That somehow the fact that the job is making games makes it special, and exempt from all of the normal moral implications of taking advantage of your staff. 

(continued in part 2…)

We were having a discussion this morning about the perils of building your business around a popular tool that you’ve made, and I thought it might make a good blog post. Really this is inspired by the impending upgrade I’m going to need to do to SmartSVN 6, despite having only just bought a SmartSVN 5 licence. It’s a nice package, especially the diff tool. And there are times when you really do want to navigate the changed files in a large tree of directories, and TortoiseSVN and the command line shell really suck at that.

My problem comes in that pretty much all of the functionality I needed from the tool has been present since version 3. Sure some of the new features are nice, and the overall application is prettier now. But mostly what has come in subsequent versions has been for me superfluous bloat. It means the installer is bigger, it means the application is slower to respond, and more complex to navigate the interface. And really what I actually would most value is if I had pretty much the exact same set of features, but a) bugs were squashed and no new ones added, b) the support for the underlying system (in this case Subversion) had been kept up to date, and maybe c) the interface was improved and streamlined.

It’s not just SmartSVN though, it’s lots of applications. GetRight, ACDSee, Word/Excel/Office, all of which have gone through many revisions, each one adding more and more features that I don’t really need or appreciate. In the case of ACDSee the later versions actually made the interface worse, which effectively removed the one feature that people bought it for – the quick and easy way to mouse-scroll through whole directories of images.

The problem here is that what the end users want is fundamentally at odds with what the business needs to do. The end user would be quite happy having the same application and just keep using it forever. But for the business that authors the application, that’s a losing proposition. They’ve made one sale, and they get no more. What they really want is to be able to sell more things to the same user. If they make a new version of the tool, they can put enough new features in to persuade the user that it’s actually a new application. The user isn’t an idiot though. They know it’s not a whole new application. Maybe they need or would like some of those new features, maybe not.

Most likely they’re buying the new version because it fixes bugs in the old version, and is better supported. I’d even go so far as to say that they’d be willing to pay for those bug-fix and maintenance updates – not as much as they’d pay for a whole new version, but something at least. Some companies, like Whole Tomato (who make Visual Assist) have a refreshingly straight-forward approach to this: you buy the application, and you get X months of updates along with it. After X months, you have to pay again a smaller amount to keep getting updates.

Even Whole Tomato though are guilty of the most annoying of the application developer’s sins (from an end user’s point of view at least) – feature creep. Every development has a Request For Enhancement list which is used as a back-log of items that might make their way into the application at some point. Anything reasonable the end users request goes on that list. And because the developer is a business, that makes its money from cranking out new versions of the application, they work through that list, pretty much endlessly. So an application which is clean, usable and decently featured in version 3 can turn into a Swiss-Army-Knife of an application by version 8; bristling with functions, features and options, which the majority of users, other that one user who requested it originally may never touch.

It seems that very rarely is the decision taken to just draw a line in the backlog and say “it’s good now, let’s just stop adding features”. Keep selling it, keep maintaining it, putting out bug-fix and maintenance upgrades, but no more features. What, I ask, is wrong with making compact applications with specific and targetted functions? Even if there are valuable features to be added, why not ask if there is value in maintaining a Core and a Premium version of the application? Or even multiple Premium versions? Find your users and target the groups who want to buy your product. Give them each their own version. Discontinue versions that don’t sell well, and maintain versions which have.

A well architected software development can maintain development on multiple branches simultaneously with not too much extra effort, certainly there is a cost but I believe there is also a reward. Give your users exactly what they need and no more, and you avoid becoming the behemoth application that is under-cut by a spunky fresh new competitor, that didn’t have all of your features but had “just enough” to be valuable, was cheap enough to develop to undercut you on price, and flexible enough due to its small size to adapt quicker and better to changing technology than you did.

Some discussion on the Chaos Engine about code reviews and check-in process had inspired me to write up a post on my thoughts on this, but on my morning trawl through my RSS feeds this morning it looks like Lee Winder has beaten me to the punch. He pretty much covers all of my thoughts on the matter, and I don’t think I disagree with any of his opinions, so hey, job done. 🙂

In summary if you don’t want to follow the link though, I’d say code reviews are a good thing, especially as teams scale and there are differing levels of skill and familiarity with the code-base. They get everyone on the same page about what is being written and how it is being built. Pretty much all of the criticisms levelled at the process are from people who have had bad experiences due to issues completely outside the scope of the process. If your code review was useless because the reviewer was being a) petulant and petty, b) disinterested, or c) unable to comprehend the issues involved; then the issue is with your team ethos and make-up, not with the review process.

I want to write up my thoughts on the ongoing IGDA debacle, but suffice it to say for now that any interest I had in re-kindling the Scottish chapter have been well and truly snuffed out.

So it seems that aside from just the usual wildlife inside the office (i.e. the ubiquitous Edinburgh mice), we have some visitors from outside as well.

Isn’t that one chubby squirrel? Tim reckons that he just does the rounds of all the windows nearby, and lives off the generosity of the locals who like cute squirrels. He certainly doesn’t seem afraid of us…

Gah. New rule for Subversion repository maintenance – run “svnadmin verify” as often as you run your off-site backup process, and arguably don’t do the backup if it fails. One of our support repositories (as opposed to a development repository which I’m a bit more paranoid about) has had a dodgy revision in for a few months now which would have bitten us had we had to restore from scratch. It looks like it was a failure while checking in a massive binary file – it doesn’t affect the day to day running of the repository, but it means that we can’t dump or load (and correspondingly can’t effectively restore from backups).

Since rebuilding the repository revision by revision is always a massive pain, I’ve done some mucking around in the guts of the repository to get around the problem. And since my Googling of the issue has been less than helpful, I thought I’d post here to give a reference for anyone else with a similar issue.

Symptom:

Running svnadmin verify on the repository results in a “Checksum mismatch while reading representation”. The output here is misleading, because it will say something like “* Verified revision 23” on the line before the error message. This means that it is in fact revision 24 which is bad. You will also find that if you try to dump the repository, it will successfully dump revisions 0 through 23, but then fail on 24. If you try to dump revisions 0:23 and then 25:HEAD like I did, you’ll probably find that the 25:HEAD revision doesn’t work.

Diagnosis:

One (or more) of the changes to files in the revision that is causing problems has a different checksum than the one that the revision file recorded at the time. So when svnadmin verify looks over the contents of the revision and recalculates the checksum it finds that they don’t match and tells you. This means one of two things: 1) the checksum recorded at the time was wrong, and the data in the revision/file is valid, or 2) the data in the revision/file is corrupt, and the checksum at the time was correct.

If the file generating the bad checksum is a text file, you might be able to look at the contents of the revision file and check if it’s noticeably corrupt. If the file is binary as mine was, that’s probably not an option. Even more so if the file is large (mine was several hundred MB).

2) seems to me more likely, so chances are the file in question is corrupt and you need to fix the data. But if 1) is the case, then all you need to do is fix the checksum. Either way you probably can’t tell at this point – so best to assume it’s gone and work from there, or at least treat it as suspicious and verify it against other sources for the data if possible.

Possible Solution:

If you’re happy to assume that file is corrupt, then you can get your repo back to a verifiable step by changing the checksum saved in the revision file to match the checksum which will be generated from the data as it is now. The data won’t change so you’ll still have to verify it manually or delete it later, but at least you can persuade the repository that you don’t care.

Process:

I’m assuming here you’re working directly with the server on Linux. I use Debian, so tools like grep and hexedit are usually available (although I had to install hexedit).  The same principles would apply on Windows, but the tools would have to change.

1) Identify the revision which is corrupt. This is straightforward – it’s the revision after the last successfully verified revision

2) Identify the file in the revision which has the bad checksum, and find the bad checksum in the revision. This is harder – the revision files (stored in /repository/db/revs) are binary, and in my case, huge. But grep is your friend here. svnadmin verify gives you the checksum that is currently recorded – this is stored in the revision file, right next to a description of the file. Here’s a grep command that searches the particular revision file for the checksum we’ve been given:

grep -e "79a1686d0dfb8618b8ccfc9eb7d74759" -A 3 -B 3 -b -a main/db/revs/24

384989609-id: 5cu.0.r24/384989609
384989633-type: file
384989644-count: 0
384989653:text: 24 75689685 293851064 294285337 79a1686d0dfb8618b8ccfc9eb7d74759
384989724-props: 24 384989543 53 0 113136892f2137aa0116093a524ade0b
384989782-cpath: /path/to/the/bad/file.exe
384989842-copyroot: 0 /

The number at the start of each line is the offset, we’ll use that soon. The cpath line is most interesting – this is the file you can expect to be corrupt. But it’s the :text: line that we need to change to get things working. As described here, (look for the section on the revision file format) this line is of the form “<rev> <offset> <length> <size> <digest>”. We don’t want to change the first 4 parameters – they’re most likely just fine. But the 5th parameter is the bad checksum, and we’ll need that in the next step.

3) Change the bad checksum to match the “actual” checksum which the svnadmin verify process is coming up with. Again, this is printed out when you run the verify. To make the change, I used hexedit, which thankfully doesn’t try to load the entire (huge) revision file into memory. You just fire it up, and press Return to enter the offset within the file to jump to. It wants it in hex, so a quick conversion turns 384989653 into 16F279D5. From there you can press Tab to switch to ASCII editing, quickly find the offending checksum and overwrite it with the new, valid checksum; then press Ctrl-X to save out the file and exit.

4) Re-run svnadmin verify. It should now successfully verify the broken revision and move on. If it doesn’t, check to see if the revision and checksum it’s failing on are the same – if they’re not then you have more broken files/revisions, and you should repeat steps 1 to 3 until they’re all gone. Hopefully there won’t be too many of them. And remember – just because your repository is now verifiable, doesn’t mean that your data is valid. All you’ve done is told the svnadmin tool that the checksum for the data you have is the same as the checksum it expects.