Sunday, February 04, 2007

Considering data conversion in an agile world

Having just typed that title, I'm already sensing a "Part 1" being appended to it any second. However, lets forge on...

We've been integrating agile development methodologies at Talis recently, and I was fortunate enough to be on one of the first projects to explore this, alongside the Scrum project framework. Bottom line - its been a great success, but its not the topic I want to explore here. I'm currently looking at data conversion, and it struck me that the approach to this may be different under an agile framework.

A 30 second google revealed Agile Data from Scott Ambler, quickly followed by this article on "The Joy of Legacy Data". Result! I'll be coming back to this in a later post, as I'm interested to apply the agile manifesto with the data conversion arena and incorporate some of this.

However, before my googling, and after one too many beers at 1am last night, I'd scribbled some thoughts on this down (using yellow "post its" - this agile stuff is getting to me). The first was that, as a product owner, my responsibilities for specifying a data conversion are similar/equivalent to those of a "functional" user story. I'm representing a customer, I'm understanding the value (in their data), and the goals they have (with this data).

Secondly, this led my to conclusion that the descrete elements of a data conversion can be written as stories - goal, value, estimate - and as such can likely be managed as such. This will allow iterations, and the developer/customer negotiation that applies to functional stories to work as efficiently for data conversion stories.

Thirdly, as both a librarian and an analyst (and having managed/analysed more conversions than I should admit), I have always felt the data conversion is the key to a successful solution. I've never tried to verbablise why, but the agile approach has made me start to consider this. Its because the value of a software application is in the data and not in the the application itself. Remove the data/metadata/content, and any application is redundent. Worthless. It is the data itself which gives the application functionality purpose and value. Its like putting no petrol in your ferarri - looks great, but it won't be able to go anywhere...

I've done both good and bad conversions (far more of the first, I hasten to add). I've always tried to add value to the data as it moves between systems. One key aspect of doing this is understanding the function/value a piece of data provides both within the system, and externally outside the system boundary (e.g. supporting business practice). A single Y/N flag can signify or support many complex business processes - or create many business problems. One of the reasons for conversion is often to remove these problems (not to replicate or add them!), and this is as inherent in the data as the function of the recieving system. The bottom line is that a poor conversion, or one that doesn't allow the recieving system to demonstrate its potential, means that all the functionality you've spent months building will not work.

One example - last sprint (for those not in "the know", thats a 30-day development cycle) we added a simple tool that allows a postcode to be clicked to launch google-maps. Useful, but de rigour nowadays. If the data conversion can't get the postcode into the right field, the story which built this is redundent. Another way of considering this is that we write acceptance tests for functional stories - it could be argued that a data conversion could effectively cause these acceptance tests to fail.

Now, we can reasonably argue that the customer could manually move this data pre/post, and this is often a very real argument with the developer/customer throwing this back and forth. Agile to the rescue! By dividing the conversion into discrete stories, the developer estimate is exposed. The customer business value is understood. The customer also has the facility to add their own estimate into the mix for their time to manually clean. The methodology ensures all these attributes in the decision are clearly quantified and exposed. We move from an argument, to an open negotiation where informed decisions can be made.

In practice, I don't see a story for every data map - the obvious 1-to-1's are no brainers. Its the one's that result in effort/pain that a story can provide a real benefit to. Which includes the ability to be more iterative in tackling these.

I'm definately feeling this is a "part one" of a longer thread, and hope to return to this as I explore this practically over coming months and discover everything I've just said is wrong...

No comments: