A basic overview of Rational’s automated testing tool, Robot, is that it records the user’s actions in a VB-like script that can be played back whenever the user requests.  This recording is done through the UI – and so it can only be done after parts of the application have been developed.  Since it will be used as a baseline for regression testing in later release cycles – it shouldn’t be written until that part of the system has been tested.  Robot scripts can be quite powerful, and the user can create ‘verification points’ to test to make sure that the data in a certain drop down is accurate, business results are correct, or even compare screenshots of the application to a baseline to make sure that the UI hasn’t changed.  While great for regression testing when no changes have been made to parts of a system, it may not be the most efficient way to test in an iterative development environment.

One of the traits of iterative development is that the customer sees the system at various stages of development.  As a result of this, there will be more change requests than in a traditional waterfall approach – ranging from small UI changes to (oops!) business processes that may have been left out of the original requirements.

The second is that only portions of the system are developed at a time.  If, at a later iteration, a module is integrated with an existing module from earlier in the development process, the old test script becomes outdated, and depending on how the two modules integrate, useless.

These may seem like small nuances, but it can become quite time consuming to maintain automated test scripts.  In a perfect world, they shouldn’t be created until a certain part of the system is stable enough to be a baseline for regression testing.  And so, after even the smallest change has been made to an application module, that entire module should be thoroughly tested again - and this can only be done manually, as the test script has likely become outdated due to changing requirements.  It is because of things like this, that automated testing can become cost-prohibitive in an iterative development environment.

And a related note (complaint?) – not necessarily against automated testing in general, but perhaps only Robot, is that there is no way to manually open a DB connection and perform SQL queries.  If module #1 has been developed, but it creates data only viewable in modules #2 and #3 (which have not been developed yet), there is no way for Robot to record that this data is being stored correctly.  Without SQL access, it becomes difficult to verify that these things – at least with an automated tool.