Imagine a computer that not only understands what you say but also says it the way you can understand. Imagine a computer that you can talk to the way you talk to humans and that responds the way humans respond. Imagine a computer that can read your thoughts and communicate with you as seamlessly as you’ve always wanted it to, since you saw 2001. OK, keep on dreaming. 2001 is in the past, yet still in the future.

But indeed a HAL of a lot has changed since 1968. Now we can build machines that can reliably understand spoken commands, whole phrases or sentences, and react accordingly: provide timetables, transfer money, book tickets, or provide assistance with any kind of problem we might have in today’s life 2.0 (man how I hate this 2.0 thing by now!). And those machines are increasingly built in a way that they use human patterns of communication, allowing for more or less free speech, interactive turn-taking, and relatively natural-sounding computer voices.

Over 9 months ago – man, time flies like an arrow (hey, we can even build machines that understand the ambiguity of this sentence; apparently already way back around 1968) – I wrote my first article on Natural Dialog Management (also check out the 11/05/2008 jam session on this topic). I promised I’d continue on this, so here I am. Today I want to write about how you can make a voice application adapt to the caller with regard to their “speaking style”, the vocabulary they use, “how they speak”. Why should you do this? Think of a doctor trying to explain what’s wrong with you. If he or she doesn’t adapt his or her vocabulary to yours, you might just as well stay home and google the symptoms yourself.

Here are some examples where Adapt-to-me (as we like to call it at Voxeo) makes sense in speech applications:

Synonyms

• If you are a provider of, say, landline telephony as well as high-speed internet, you might have callers calling into your helpline saying “I have problems with my Internet connection” at your first How-Can-I-Help-You input state. Your system might confirm this by saying “I understand you have problems with your DSL, correct?”. Your technology to provide internet might be DSL – but does your customer necessarily know that? How could she respond? Maybe with saying “No, Internet!”?
  Ouch…

Number patterns

• Ever had the experience of giving out your phone number over the phone and hearing it back from your interlocutor in a way you didn’t even recognize your own number anymore? “My number? That’s six two nine three nine oh four.” – “OK, I’ve jotted that down, that’s sixty-two, ninety-three, nine fourteen?” – “Hang-on… let me think… err, yeah I think that’s it.”
  Ouch…

Date patterns

• How do you say the expiration date of your credit card? If it was “12/12”, would you say “twelve twelve”, or “December twelve”, or “December two thousand twelve”, or …
  No ouch this time. This is just to demonstrate that there are numerous ways to speak dates, and using the same pattern as the caller when repeating their input again can help improve intelligibility of the system, thus cause less frustration, thus fuel acceptance of the overall application, thus increase revenue…?

So I say you can have the computer say “internet connection” instead of “DSL”, and “six two nine three nine oh four” (not even “six two nine three nine zero four”), and even “twelve twelve” if that’s what the caller is inclined to say (maybe hastening to add a “that’s December, two-thousand and twelve”, just to confirm you have fully understood your caller). You will say: “How”? Let me explain.

VoiceObjects allows you to store the pronunciation of an utterance in a variable, along with its actual value. This is done through the grammar that enables the speech recognizer to understand the caller in the first place. This value is called the pronunciation value. There is no fixed format for how this value should look like; it is completely up to you. How to hand this value back to VoiceObjects Server from within a grammar is simple: you add it to the return value for the slot that is filled by the corresponding utterance, separated by a double-pipe (“||”).

Example:

When the server detects this “||” symbol (which is configurable through our media platform driver concept, by the way), it will parse the actual value out of it (“DSL”, as this might be the internal value required for further processing) and assign it to the variable, parse the pronunciation value and assign it as well. By the way, if you’re interested in this value during processing, you can retrieve it via the PRONUNCIATION(RefID) function provided by the Expression object.

What you do with this pronunciation value is straightforward, too: you hand it over to a formatting algorithm (via our Formatting Bus), which takes this pronunciation value (along with the “real” value, which is actually not needed for speaking the variable value back) and uses it to come up with the pronunciation when repeating the value in an output. Note how the grammar, in the above example, returns “internet_connection” as the pronunciation value; this assumes that there is a prerecorded prompt saying “internet connection” as the problem category. Your formatting algorithm would thus probably need to return “internet_connection.wav” as the audio file to use for playback. In fact, for this example you don’t even need your own formatting algorithm. The predefined formatting types utilize the pronunciation value instead of the actual variable value anyway. So choosing, e.g., TTA – Files or TTA – Complete as formatting types for your Variable object will make the platform use “internet_connection.wav” right away. Nice and simple.

Let’s have a look at the number pattern example now.

First, your grammar must be built in such a way that it can recognize single digits as well as number blocks. Usually, rules that match “one” up to “ninety-nine” suffice. The rest can be nested using smart grammar rule structures. In the tags that compute the value of what was said (as opposed to the words used), you need to add logic that also builds up the pronunciation value as the caller speaks (or rather: as the ASR engine computes the result). As an example, if the caller in fact says “sixty-two ninety-three nine oh four”, the slot return value computed by your grammar rules might be “6293904||62 93 9 oh 4”, which gets parsed as “6293904” for the actual variable value and “62 93 9 oh 4” for the pronunciation value. Your formatting algorithm might make a sequence of “62.wav 93.wav 9.wav oh.wav 4.wav” out of this. In fact, you could just as well use a predefined TTA algorithm for this again, e.g. TTA – Words, and it will do the job.

Last but not least, our famous sample application Prime Telecom, a telco self-service portal coming in three channels (voice, text, mobile Web), provides a sample implementation of Adapt-to-me with the credit card expiration date example I described above. Go check it out today! You can get all the software required to run this sample application for free at http://developers.voiceobjects.com. Go and impress your boss with what VoiceObjects can do for making your phone applications a much more pleasant experience, and your customers much happier. (Or maybe you ARE the boss? But hey – this mission is too important for me to allow you to jeopardize it…)

Oh, and if your boss tells you to implement this within your existing VoiceObjects app, check out the Input object documentation of our Object Reference (search for “pronunciation value”).

In short, you need to handle sets of test data, each set representing a certain test case. Of course, there are several options to deal with this, but as of VoiceObjects 7.4, you now have a very elegant solution at our fingertips: The new expression function APPLYCONFIGURATION.

What does it do? Let’s have a look at the inline documentation in the Expression editor:

APPLYCONFIGURATION (configurationXML) – Applies the assignments defined in configurationXML. The XML format used is the same as for application defaults.

Application Defaults

Have you used the application defaults functionality before? If not – it’s simple: It’s about initializing selected variables, layers and collections on the service level. The Service object references an XML configuration file in the Configuration URL field. This configuration XML file will be loaded whenever the service is (re-)deployed. (For more information, check out the section on Application Defaults in the VoiceObjects Deployment Guide.)

The primary use case for Application Defaults is this: When working in multiple environments such as, say, a development, a test, and a production environment, each of those will require some unique configuration settings. For example, database names and credentials might differ, resource locator paths, and any other external settings. By “outsourcing” the initialization of the environment-dependent variables to the “application defaults” configuration XML document (which is bound to the service object, not to the project), the project definition itself becomes agnostic of the environment and can hence easily be taken from “dev” to “test”, and from “test” to “prod”, without applying any changes to the project.

Click to enlarge

For an example of a valid configuration XML file, scroll down to the bottom of this posting. In a nutshell, a configuration XML file references any number of variables, layers, and collections in a given project (by reference ID) and defines their initial values.

In-Session Configuration

Now, VoiceObjects 7.4 takes this concept one step further and makes the same mechanism available on a per-session basis: You can do bulk assignments of variables, layers and collections in a single step within your call flow definition, applying different sets of values for each and every call. Of course, this comes in very handy when you need to manage test case data.

Let’s have a look at our Prime Telecom demo application. It supports 3 different languages and 2 different customer types. For each of the resulting 3×2 = 6 combinations, we need at least one test case.

In the previous version of Prime Telecom, these test cases were handled in the traditional way: Within the Preprocessing sequence of the main Module Prime Telecom Portal, a Connector object invoked a JSP, providing the current language and the customer status as request parameters. As response parameters, the Connector’s parameter set contained each and every variable and collection that needed to be initialized – the customer’s postal address, email address, payment information, current tariff, subscribed tariff add-ons, available tariff add-ons etc. Quite a few parameters had to be maintained. And whenever a new parameter had to be added, it had to be added both in the JSP implementation and to the Connector object’s parameter set. Also, the maintenance of the test data in that JSP was cumbersome at best.

Not so any more.

The new implementation of test case data handling in Prime Telecom relies on test data being organized in configuration XML files, each file representing one test case. In Prime Telecom, these files are named configuration_de-DE_platinum.xml, configuration_de-DE_silver.xml, configuration_en-UK_platinum.xml etc.

In the Preprocessing sequence of the main Module Prime Telecom Portal,

1. a Connector object reads the configuration XML file (via http get) for the current language and customer status and assigns its content to a variable;
2. this variable is then used as the argument of an APPLYCONFIGURATION expression, setting all required variables and collections at once.

The beauty if this solution is that there is only one place to maintain the test data: In the configuration XML documents. When adding more parameters, only the XML documents need to be adapted; the Connector implementation (in our case, some Java code) and the Connector object’s parameter set remain unchanged. Also, the configuration XML documents are much easier to read and hence to maintain than the old JSP.

Of course, there are more use cases to APPLYCONFIGURATION than “just” handling test cases. For example, a hosted service provider could build application templates which become adapted to each customer’s requirements using this mechanism. Also note that, using VoiceObjects’ web service interface, much of the necessary handling could be automated, creating easy-to-use web front ends for end customers.

Example for a valid configuration XML document

This example shows how two objects are being initialized - the variable with the RefID CustomerBaseTariffName, and the collection with the RefID CustomerPaymentSettings. Note that the <type> nodes are optional; also note that collections need to be masked by <![CDATA[ ... ]]> sections.

<?xml version=”1.0″ encoding=”UTF-8″?>
<configurations>
  <configuration>
    <referenceID>CustomerBaseTariffName</referenceID>
    <type>variable</type>
    <value>Individual Plan</value>
  </configuration>
  <configuration>
    <referenceID>CustomerPaymentSettings</referenceID>
    <type>collection</type>
    <value><![CDATA[
      <root>
        <row>
          <col name="type">Visa</col>
          <col name="number">4140040912440644</col>
          <col name="expdate">0210</col>
        </row>
      </root>
    ]]></value>
  </configuration>
</configurations>

As you may know, VoiceObjects 7.4 has been released and is available for download on this developer portal.

Now, I wanted to see what’s in VoiceObjects 7.4 for developers. In particular, I was interested in exploring some of the new expression functions – there is large number of new functions in the realm of date and time handling, string operations, regular expressions and more, plus some other more VoiceObjects-specific functions. My plan was to adapt our Prime Telecom demo application to leverage some of the new VoiceObjects 7.4 functionality. Of course, the existing “legacy” Prime Telecom application works fine in VoiceObjects 7.4, but then … at the end of the day it’s a demo application, so I thought it should stay up-to-date, best-practice-wise.

I started reviewing the existing Connector and Script objects in Prime Telecom, assuming that these presented the lowest hanging fruits: If I could replace some of them by mere Expression objects, using the new functionality, the gains in application maintainability and scalability would be obvious. And indeed, I did find a few objects that had become obsolete:

• Three Connector objects that had called custom Java code to validate the format of email addresses, credit card numbers, and expiration dates, could be replaced by simple Expression objects using the new MATCHESREGEXP function. This function checks whether a value matches a given regular expression. No more Java code to maintain here! Check out, for example, the new Expression object Is Email Address Valid in the new version of Prime Telecom to see how it’s done.
• The Script object First Day Next Month had contained some JavaScript logic to calculate, well, the first day of the next month (relative to the system date). With the new expression functions LASTDAYINMONTH, ADDDATE and CONVERTDATE, this Script object could be easily replaced by Expression objects. There are quite a few more date and time related expressions, like NEXTWEEKDAY or TIMEBETWEEN that should help you implement most conceivable tasks in date and time arithmetics.
• The Script object Monthly Rate for All Active Addons contained JavaScript code that iterated over the prices of all tariff add-ons currently subscribed to by the caller. Whenever the caller subscribes to a new add-on, the total monthly rate has to be re-calculated, so this Script object had to be re-executed. Now, with the new Expression function ITERATE, it was easy to replace this script by a few expressions: First, a Collection object is initialized with all prices of the current add-ons (Expression object Assign List of subscribed Add-On Prices). Then, after resetting the “total monthly rate” variable , another expression simply iterates over all rows in this collection and sums up the prices (Expression object Sum up Prices of all Active Add-Ons).

After getting rid of a few Connectors and Scripts, I reviewed the existing Expression objects in Prime Telecom. I found some implementation details that looked a bit cumbersome in the light of the new possibilities. Hence, I tried to replace them with new expressions, lowering the total number of objects used and trying to make the code more maintainable at the same time.

• First, I investigated the new string operations. For a start, using the new functions RIGHT and LEFT instead of MID where appropriate helps keep the number of objects down.
• The new function COUNTOCCURRENCES made several other expressions in Prime Telecom obsolete. It counts how often a certain substring appears in another string. In combination with the XPATH expression, it helped implement a very efficient way to count available tariff add-ons: Inspect, for example, the Expression object Add-On Row Counter.
• Note, by the way, that the XPATH expression now takes a third argument returnAsCollection: This allows returning a list of XML nodes not only as a variable containing blank-separated strings (returnAsCollection=false), but alternatively as a handy Collection object (returnAsCollection=true). This makes the result of an XML query much easier to handle, as you can see when you inspect the Calculate Subscribed Add-Ons Sequence object.
• In the same sequence, I found the new ITERATE function to be useful. I use it in the Iterate over remaining Add-Ons Expression object to create a comma-separated list of the names of subscribed tariff add-ons. ITERATE can make the (much more powerful) Loop object obsolete in cases when you just need to iterate over an expression to perform some simple logic.
• The new VALUESUBSTITUTION function was instrumental in minimizing the number of expressions in the context of managing the headlines in the web and text channel. This function extracts a piece of functionality that was previously only available in the context of the Formatting object’s Value substitution field: It searches a variable in a Collection object and returns a lookup value – which can differ per language. This comes in very handy in multi-language applications. Check out the expression Headline to see how the (language-dependent) headlines in Prime Telecom’s web and text channel are implemented. 

There are many more new expressions to explore; some of the most powerful ones, like APPLYCONFIGURATION, deserve dedicated blog entries just to explore their capabilities.

Let me conclude with two more nice catches that have an impact on the Prime Telecom implementation:

• The Channel filter is now also available on the Menu Items in Menu objects and on the Correction Mappings in the Confirmation object. This makes it much easier than before to create multi-channel applications with different sets of menu items across the different channels. For example, you might want to offer an “Address Change” self-service in the text-based channels, but not in the voice channel. For an example in Prime Telecom, check out the Confirm Credit Card Confirmation object: The “Correct both” (both the credit card number and the expiration date, that is) Correction Item is available only in the voice channel.
• In the web channel, fields in web forms are now automatically pre-populated with the according variable values. This works for text fields, radio buttons, and lists. Note how the Prime Telecom web forms no longer “forget” your input when correcting, for example, credit card details.

If you want to reproduce my findings, I suggest you download your copy of the new version of the Prime Telecom demo application, install it in your VoiceObjects 7.4 Developer edition and inspect it in detail.

If you are a developer interested in evaluating, playing around with, or just getting to know VoiceObjects, a good way to start is looking into the tutorials and demos that come with it.

In particular, if you’re interested in the multi-phone-channel capabilities of VoiceObjects, or if you just want to explore some of the more advanced implementation concepts of the VoiceObjects platform, you should have a look into the Prime Telecom demo application. You find it in the Demos&Templates section of the developer portal.

Very recently, we have added a few video demos that show how the Prime Telecom application looks&feels like in the different channels. Watch these videos to see what you will be rewarded with if you decide to download and install the Prime Telecom demo.

First, watch a short demo of the Prime Telecom application in the text channel. The same “Enter new credit card data” self service can be invoked on the mobile web channel and as a voice call. For a high-level backstage view of the service implementation, watch this video.