Background
The inclusion of a chatbot experience is something most banks are beginning to adopt as part of their customer care strategies. When done right, they can reduce wait times, reduce phone calls, and lead to more resolved cases users may have.
The Bank of Montreal is planning a launch of their chatbot experience. To prepare for this, I helped their team run some moderated usability testing with a chatbot prototype to get a better understanding of usability issues and possible improvements
🎯Goals
• Uncover any usability issues in the chatbot experience
• Determine how confident users feel with the chatbot to solve their problems
• Determine how satisfied users feel with the chatbot and the solutions they offer
• Understand the user’s expectations towards the chatbot and uncover if there are any opportunities for improvement
Format
Moderated usability test via usertesting.com. 60 minute sessions.
Participants
• N = 6
• Age = 26 - 65
• All participants who qualified for this study were in their 30s ( 4 male, 2 female)
• 3 BMO customers, 3 other Financial Institution customers
• All utilize mobile banking at least once a month
• All respond that they would be likely to use a chatbot experience
• Location = Canada
Participants were given access to a prototype of the chatbot experience via an Invision prototype. Tasks and scenarios involved finding the chatbot, and making a variety fo queries utilizing both pre-populated options as well as typing to find specific options.
Testing was done with an Invision Prototype with masked branding
👩‍🔬Insights
How users reported getting help:
• While the primary focus of this study is to asses the chatbot prototype from a usability perspective, users were also asked to explain their general use cases for chatbot.
• Popular user behavior is to try and fix the problem themselves first and explore options at the area they are having issues with
• There is an expectation that generally most bank functions are easy to complete by default and can be resolved through trial/error or searching on the internet (a common user behavior for troubleshooting)
• However, having access to a bot would be useful to users who expressed this sentiment when they are experiencing irregular occurrences like a charge they don’t recognize for example.
âś… Feature idea: Possibility for Chatbot Quick Links
When participants described their own self-serve patterns, 6/6 participants suggest that they would likely be navigating themselves to the area in the app they are having problems with. For this reason, it is also a possibility to have contextual prompts for chat support in some of the main action areas
Chatbot Discoverability
• All users were able to notice the chatbot placement on the account overview section
• The other place users suggested they would look for chatbot would be somewhere in the “more” menu alongside other help & support materials
3 / 6 users mentioned however that the placement in the prototype did feel either like an ad or that it felt much more larger than it needed to be. These users suggested maybe wanting to see something smaller and in a corner, which is a common placement for chatbots in other product designs.
Chatbot Expectations
All participants in this exploration have had some form of experience with chatbots
Participants generally expect some generic options as well as the option to talk with a live agent
There is a sentiment among participants that they expect a chat feature to be present, especially with a competitive mobile banking app. Especially as an option before calling, which participants don’t prefer to do right away
There is mixed feelings with chatbot as a whole
While participants expect chatbot to be present, they also don’t have a high amount of confidence in the bots ability to answer questions
Participants often feel like they can waste time going in a loop with the bot not answering their questions and then having to call in
In this case they suggest it would be easier for them to call in by default
Every user expects there to be access to live agents as they feel without this, the chatbot experience may be very shallow and not helpful for their needs.
Because the first iteration of chatbot does not have live chat, it is recommended we explore methods to help provide users with contact information at various points in the conversation as alternatives to solutions that the bot provides
Chatbot Usability
From a usability perspective, all users were able to utilize the chatbot in the prototype to complete both tasks. Users used a combination of tapping on suggested responses or typing.
Majority of users utilized the options provided and found it very useful for the chatbot to provide options to them
2 users began typing only when they did not see the option they were looking for. Once they did see their desired option, they utilized the provided options.
A common pattern that came up for users is having issues going back in the chatbot history or selecting a different answer
some users tried to type in a command to return them to the previous options they had
other users tried playing with the prototype to return to the previous options they had
We should explore natural & convenient ways a user can start over with the chatbot or explore previous options they were looking at so they don’t feel like they need to close the experience and open it again to reset if they chose the wrong option
Navigating back to the chatbot
Of the cases where users left the chatbot to complete an action, all were able to understand that hitting the arrow button in the top left of the navigation bar would have them return to the bot
Users typically expect that the chatbot maintain their chat history and provide them with additional options to continue or restart
On return is when users would suggest they would expect any sort of feedback options.
Users typically expect some sort of scale or ranking that is easy for them to complete
6/6 users missed the feedback option within the chat
How can we make feedback more organic and part of the chat? We can consider using contextual prompts when the user returns to the bot with a simple scale.
Expectations for unrecognized inputs
When asked how users would assume the bot would behave if it received an input it did not recognize, all 6 users gave a similar response where they assumed the bot would enter a loop trying to get the user to clarify until ultimately providing contact information for the user
Expectations for how long the loop would go was mixed, generally ranging from 1 more try to 3 tries.
Language & Immersion
When asked, all users generally felt that the language used in this chatbot experience was pleasant. Users had a strong sentiment that the wording used in a chatbot experience should be too the point and also with an overall positive tone. Users suggest they generally know what they are looking for and do not want to spend too much time reading to find it.
For 1 user, they felt that “let’s go” felt too vague as a CTA and would prefer if the bot was more explicit in terms of the direction they were taking users
For the other users, they generally felt as “lets go” would direct them to the area they would like to complete a task in.
When it came to immersive chat elements such as animations, alert noises, and other feedback mechanisms, users expressed liking the knowledge that something was happening behind the scenes and that their input was being recognized.
Alert noises were less important for users when it came to only the chatbot experience but became more important when it came to a live chat experience as they could know when they received a reply (assuming they put their phone down to do other things).
As an interesting note, one user who spoke French gave their thoughts on previous experiences with chatbots that converse in French and issues they had with the informal tone that was used by the bot.Â
Suggestions for improvement
1. Additional Help Options
Half of the participants expressed not seeing is more help related options that seek to troubleshoot specific issues with each section as opposed to just navigational related help items .
This impacts overall chatbot confidence as users feel like the chatbot experience may not cover all of the reasons why they may be needing help
How can we make our chat feel more supportive? Maybe instead of having a catch all option to view FAQs, we can surface some of the contextual help related FAQs related to a subject as options in the bot?
2. Inclusion of Live-chat
Majority of participants feel that a strong component of a chatbot experience is the option for a live agent or live chat feature.
For some users this was ranked the most important feature of a chatbot experience
The biggest frustration that users bring up over past chatbot experiences is the fear of going in circles with the bot not providing any helpful resources and the user feeling like they could have saved their time by just directly calling.
This is less than desirable for users because they already suggest they want to chat to someone before calling as it is faster and more convenient.
✨ Conclusion
Overall, we found that in these sessions the chatbot experience performed well from a usability perspective. Users were able to recognize where to find the chatbot and were able to complete the provided tasks using a combination of typing and selecting provided options.
Users understood how to navigate back to the chatbot as well after leaving. The chatbot experience overall is what users expected
The only frustrating action for users was selecting previous chat options and traversing the chat backwards incase they made a mistake.
There was a general willingness to try and use the chatbot
When applicable, users suggest they would prefer to chat with customer support as a first step before calling in
Chatbot is also something users expect to see in general, especially within a competitive mobile banking application
Chatbot confidence was mixed
While users felt like the chatbot could help them for very basic tasks, users did not feel like the chatbot would help them for more irregular occurrences - which was a reason provided for when users did actually suggest they want to contact customer service
Some users also felt that the chatbot lacked overall troubleshooting options or options to help them resolve problems instead of just finding where a specific function existed
Live-chat is very important to users
For some users, this would be their entire reason to utilize the chatbot experience
Users have mentioned having past experiences where they went in circles with a chatbot and found it to be easier had they just called in
There is a need to better understand why users are going to that chatbot in the first place. The use of a chatbot is proven to be something users expect to see, however the nature of how it guides users to self serve needs to be explored.
Limitations
The scope of this research involves:
• If users can recognize the chatbot in the navigation
• what users expect from a chatbot in BMO
• how users feel about the chatbot experience
• barriers to chatbot usability
• things they would like to see in the chatbot
This research cannot determine:
• Who the chatbot target audience is
• What the order of chatbot response options should be
• Validating the chatbot concept as a whole
Another limitation is that the audience recruited was digitally active users in their 30s. Could be worth branching out to younger/older audiences to gauge impact
