As the Beatles Sing: Come Together, Right Now for Integration Testing

I am going to say it. In my experience, coming together for integration testing gets little love. In this post I am going to explore some of the reasons for this and what can be done about it. I admit that I don't have all of the answers. I often feel like the projects I have seen could do a better job with this. However, if you have been reading this blog you probably realize that these posts are sometimes a way for me to 'think out loud' to see if there are better ways to do things in the future.

Definition From Reg. Guidances

First, what is integration testing when it comes to medical device software? Let's look at the big daddies of medical device software regulatory guidance, namely, IEC 62304 "Medical device software – Software life cycle processes" (not free, link to purchase) and from the FDA "Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices" and "General Principles of Software Validation". The FDA's General Principals of Software Validation defines Integration Testing as:

Integration level testing focuses on the transfer of data and control across a program's internal and external interfaces. External interfaces are those with other software (including operating system software), system hardware, and the users and can be described as communications links.

I agree with this definition but it isn't very helpful. I think you could argue that most testing fits this definition. (The other FDA guidance says you need to do Integration Testing but does not define it.)

Fortunately, IEC 62304 goes into more details. First, you need to have information in your software development plan on the integration of software items in general including SOUP (software of unknown provenance)/ OTS (off the shelf) software and also Integration Test planning.

The MANUFACTURER shall integrate the SOFTWARE UNITS in accordance with the integration plan.

The MANUFACTURER shall verify and record the following aspects of the software integration in accordance with the integration plan:
a) the SOFTWARE UNITS have been integrated into SOFTWARE ITEMS and the SOFTWARE SYSTEM; and
b) the hardware items, SOFTWARE ITEMS, and support for manual operations (e.g., human-equipment interface, on-line help menus, speech recognition, voice control) of the SYSTEM have been integrated into the SYSTEM.

There is one note in the same section that confuses things though.

NOTE 2 It is acceptable to combine integration testing and SOFTWARE SYSTEM testing into a single plan and set of ACTIVITIES.

I think this could be read as one need not do integration testing if one is doing system testing. This is not the right interpretation though. Integration testing is important in its own right.

Ideas for Doing Integration Testing

Let's assume you have automated unit testing that runs on the build server as you build the software. That unit testing, if done right, ensures that you each code file is implemented properly. The next step is combining the units and performing integration testing. In an Agile development process the creation and integration of units happens frequently. If doing sprints, this integration testing will likely happen every sprint. Automating the integration testing will simplify this repeated operation. How to do the automated unit testing is system dependent. If you have a UI framework like QT integration testing can be done on a build machine using something like Squish. This will allow one to quickly iterate and find any integration issues. For mobile apps automated integration tests can also be done. However, for embedded systems without easy hooks for automating testing manual testing may be the only option. A strategy which can work is to have a script of manual integration steps covering core functionality to run through with each story before committing to the master branch.

Documentation of Integration Testing

If the integration is being run automated, each build will generate integration testing results. Similarly, if the tests are run manually prior to the commit to master then this will generate results. Do these need to be kept? This is where the original integration test plan comes in. In that plan one would need to decide when in the development process the results are worthy of keeping. Early in the process the results will likely not be very useful because only a few units will be available per integration. Likely the plan will not need one to keep these results. As more is integrated the results will be more useful. These will likely need to be archived along with any issues found. The final clean integration results before code freeze and start of system testing will most definitely need to be recorded.

Come Together

Not sure anyone understands the Beatles song Come Together ("he got toe-jam football", huh?). I am not sure this blog post will make us understand coming together on integration testing a breeze either but it did help me consolidate a few ideas.