Added by Johannes Brodwall, last edited by Erik Drolshammer on Sep 08, 2008  (view change)

Labels:

Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.

Responsible: Johannes Brodwall

I have never written a piece of code that worked the first time I tried it. This has turned me into a fundamentalist when it comes to automated testing. I don't want my users - or my testers! - to spend their time dealing with my bugs.

Over the last year, I have been able to extend the our build system to cover automated tests of requirements and full scale integration test of a fully deployed system. The system automates all our testing and gives us a turn around to (partial) production of 2-3 weeks.

In the session, I plan to describe the elements of the tools that we use to support testing:

  • Continuous integration with a custom system
  • Functional tests using FitNesse
  • Automatic run of recorded production data after every build
  • Systematic run of large amounts of recorded production data before release
  • Automatic parallel execution of every operation in an isolated staging environment
  • Automatic comparison of test runs with historical production
  • Monitoring of all environments

I want most of the session to be a discussion on:

  • How much testing is enough?
  • What are the perils of too much or too little automated testing?
  • How can our approach of using canned production data be used in another setting?

Moderator: Johannes Brodwall, (future) Chief Scientist at Steria

I´d like to shift the focus a bit more on how a more enterprise pojo approach like qi4j, EDR and DSL extensions of shared object impact the mindset, methodology and toolset (including but not limited to testing) in the CASE discussions

POJOs were definitely important in order to get this concept to work. Automated testing all the way to production means that we cannot rely on a complex application server. We also have to limit the number of separately deployed components in the solution and rely more on in-process communication. Unit testing is much easier if we stop using EJBs.

I don't have any experience with qi4j or DSLs in this solution. We use a concept like EDR (if I can remember what that means!), but it's a rather inconsequential and boring part of Getting it to Work.

I respect the focus of the conference, but I've had to go beyond looking at what happens inside a JVM to succeed. Instead, we had to focus on simplifying the infrastructure and integration to where the "architecture" has nearly evaporated over the last two years. I believe this is a good thing, and that fancy Java technology can work against the goal of a simpler solution. It start with saying "we don't believe this is a real requirement" when we're presented with something that sounds like we would need a bus.

  • How do you keep the recorded data and the new solution in sync? (How to avoid maintenance hell, so to speak...)
  • How did you automate the tests covered multiple deployment units? Plain, old bash-scripts?

Given that the system in question have domain objects that should behave differently in different contexts; how would COP and similar approaches effect the chosen architecture and test strategy?

Great questions, Erik.

My first realization was that there was no way we could do this without simplifying. The first step was ditching the application server. The second was to reduce the number of deployment units. My system currently have three deployment units that all run on the all application nodes, plus that database deployment scripts. As you suspected, this is tied together with a boring batch-script. (Boring is good)

We have also reduced the number of places where we use variable behavior. Ideally, we just replace the persistence layer with an in-memory variant during unit test and functional tests, which is a one line configuration difference. This means that we hardly don't use mocks or similar techniques any more. (Other things that have variable behavior are dynamically configured based on the persistence layer, for example, if the datasource is hsqldb, we set hibernate.hbm2ddl_auto to create, if it is Oracle, we set it to validate)

The integration/system/full scale tests runs the whole system wired together identically to production. That's of course the whole point.

We don't have to deal with multiple versions of production data. The production system gets data from our customers using a published protocol, so this never changes. This is more luck than skill, of course.

Posted by Johannes Brodwall at Sep 04, 2008 19:06 Updated by Johannes Brodwall


My first realization was that there was no way we could do this without simplifying.

A good discussion is if these simplifications are applicable for a broader range of enterprises, and if there are ways to solve the real problem by using new technology and different approaches. I believe that will be a very lively discussion.

This is indeed going to be an interesting discussion.

The question isn't whether the simplifications we've made are generally applicable, but how important technological simplicity is as a design force. Often simplicity and flexibility are at odds. I believe simplicity is much more important.

Based on the discussion, I am considering adjusting the topic slightly:

"A voice of dissent: The need for simplicity

Mechanisms like AOP and qi4j are cool, but they make our systems come complex and harder to master. The fact that systems are wired differently between test and production lets bugs escape testing unnecessarily easy.

I want thorough testing to be considered a primary attribute of good software architecture, and I am not sure that the current trends support this."

One simplistic way of defining enterprise is organisations where it is not possible to chase down your simplicity strategy

I would rather define an enterprise as an organization with a potentially profit-making business of on interesting scale. I work from the assumption that interesting businesses mustn't necessarily have a lot of complex problems and complex problems mustn't necessarily be solved by complex technology. :-P

My experience is limited, but based on it, I think we create most of our problems by assuming that the solution must be complex, and almost no problems by assuming that the solution can be simple.

As a dissenting voice, I don't think we need to agree on the conclusion. Do we agree that it is an interesting topic?

Sp any company making tonns of profit is an enterprise - interesting twist

And I´d say that this discussion proves that the topic is interesting

All content on this wiki is licensed under a Creative Commons Attribution 3.0 Unported License.