Our partner,
NuEcho, have started a beta program for the latest addition to their tools portfolio:
NuBot, an automated test platform for functional testing and load testing of IVR applications. By the way, NuEcho are still accepting participants in their beta program; so if you’re interested, tell them!
Curious to see how NuBot can be used for testing VoiceObjects applications, I enrolled in their beta program and got started right away. I was delighted to see that the NuBot ITE (Integrated Test Environment) client comes as an Eclipse plugin, so it fits in nicely with VoiceObjects Desktop for Eclipse. Here’s a little screenshot from my Eclipse Perspective selection popup:
What NuBot does? Let’s listen to NuEcho: “The NuBot Platform is a complete and integrated testing infrastructure which allows for testing interactive voice response (IVR) applications. It can be applied to multiple types of applications, whether they use voice recognition or not. The NuBot Platform uses an open-source telephony platform and supports a wide range of telephony standards (SIP, T1, RNIS, analog, and so on). The system’s architecture is as follows: by way of the NuBot plug-in client, the user’s workstation connects to the remote robot server over an IP network (RMI), which in turn processes and executes incoming or outgoing calls.”
Remember the
VoiceObjects LoadTester? The conceptual difference between LoadTester and NuBot is that LoadTester loads VoiceObjects Server directly (sending http requests in place of the VoiceXML browser), while NuBot allows for end-to-end, over-the-phone testing that involves the entire software stack including the IVR. Both tools have obvious use cases in the testing process.
Now, what does it take to create and run test scenarios for an existing application?
1) Application Instrumentation with DTMF sequences
First of all, your IVR application must be instrumented. To each input state that will be part of a test scenario, you add a unique, fixed-length DTMF sequence. For instrumentation of our dear old friend, the
Prime Insurance demo application, I decided to add the sequence “C001″ to the main menu, “C300″ to the Car Insurance welcome prompt, etc. (In case you didn’t know –
DTMF keys comprise not only the well-known *, #, and 0-9, but also the characters A-D).
To make the DTMF instrumentation a smooth experience, I created a new formatting class “DTMF Sequence” for the
VoiceObjects Formatting Bus. It takes a sequence definition such as “C1#” and maps it to the according files: c.wav, 1.wav, hash.wav. Simple. I also made sure I can (de-)activate the instrumentation through a single variable, either by setting it in the initial URL, or by listening, early on in the Prime Insurance application, for a hidden DTMF command that only my NuBot test script knows about. If you want to know more about this Formatting Bus implementation,
let me know.
2) Drawing the call flow
Now let’s move on to the NuBot ITE. After creating a new test project, you first need to create a “call flow”. This is a very simple mapping of your application’s input states and their transitions, identified by the DTMF sequences which you defined and created in step 1. The following screenshot shows how I sketched out the “Car Insurance” Module in Prime Insurance as a NuBot callflow. For example, the initial prompt in the main menu has been extended by the DTMF sequence C001; and the “Ask for car” input state by C301.
3) Defining a test scenario
Now that NuBot is aware of the basic structure of your IVR application, you can go ahead and create test scenarios. The following screenshot shows such a scenario that drives and tests the Prime Insurance application, simulating user input via dtmf keys and speech input. For the speech input (“Ford”, “Focus”), I created two short voice recordings and added them to my NuBot project. When executing the test, NuBot compares the DTMF sequence played after each step by the application with the DTMF sequence from the call flow definition; in case of a mismatch, an error is reported and the test is aborted.
4) Executing test script
Finally, you create a “test descriptor” where you define which test scenarios should be executed, which phone number is to be called, how often to repeat tests, how many of them in parallel … in short, you can schedule both functional tests and load tests, and I guess, at some point, also schedule monitoring tests (running 24×7 on a regular basis to monitor a production system, performing functional end-to-end tests).
After running the test, the results are fetched from the NuBot Server and can be analyzed locally. You get summary statistics on test success/failure; you’ll see which input states caused test scenarios to fail; and you’ll get detailed statistics on response times. The following screenshot shows the response times from my first successful test execution of Prime Insurance’s Car Insurance module.
I found NuBot easy to master and a very powerful addition to my automated testing portfolio. I can only recommend to get your hands on it and try it; it’s about time that we take automated testing more seriously in the IVR application business.
RSS FEED