Last month we began our backstage tour of Illumina’s work by exploring the world of graphic design and what it takes to create a world-class look and feel for elearning. We introduced you to our infographic of our team approach to content creation. This month, we’re going to look at quality assurance, and the work we do to ensure things work perfectly in your elearning programs. Our blog editor caught up with Ashley Shaffer, Illumina’s magnate of precision and detail to hear her story of quality assurance work. 

Quality Assurance: The Nuts and Bolts

Jean: Quality Assurance is a very detail-focused job. Do you consider yourself to be detail-focused in general? What’s a non-work example of something that shows that?

Ashley!

Ashley: I think I’m a selectively-focused detail-focused person. I like looking at both the big picture, and the details in sections. Outside of work, I think this is best shown with jigsaw puzzles and puzzle games, so I’m still solving problems in my free time.

With jigsaw puzzles, you have a big picture you’re trying to solve, but you have to complete it by putting it together one piece at a time. While I can see the goal on the box art, I need to check each puzzle piece and make sure it fits next to other pieces. Some of the parts are easier, like pieces with a border go on the outside, while pieces with similar color often need to be grouped somewhere in a particular area, I won’t know until I can get the similar pieces together, so I can focus on a section.

With puzzle games, there are often stages or levels. So while I’m working on the big goal of finishing the game with a 100% completion, I have to tackle the puzzles one at a time. I have to solve the lower levels and easier puzzles first so I can figure out how the developers designed the game, and learn about recurring themes, so I can notice them later in the harder levels. With games, you can also take a break between each puzzle, so you can come back and refocus on the next one with a clearer mind, and you don’t have to worry about doing everything all at once.

Proofreading crossword puzzle

Jean: Solving puzzles is a great metaphor for the QA process. What are the steps you take to review an elearning project to ensure quality?

Ashley: I grew up with the saying “a stitch in time, saves nine”, so I prefer to look at the materials as early as possible. Starting with the “copydeck” or storyboard, I quickly look to see if there’s anything that catches my eye. For example, if I notice some inconsistencies with periods in bullets in the storyboard, it’s better to flag it and update the file, so the developer can input the right text the first time, than to find it later, making the developer go back and fix the text they already put in and published. And later, if more items are caught in the Alpha version, then we’ll have less to change or update in the Beta, which is closer to our final release date when everyone is feeling a little more rushed.

It’s also good to run through a program multiple times, rather than trying to catch everything all at once; this is where the selectively-focused part comes in. There are so many different things going on at a single time that it can be hard to focus on everything all at once. If you get overloaded, then mistakes start to fall through the cracks. If there’s audio: is everything clear and grammatically correct so learners can follow along? Did the narrator read every word correctly in the script? Are the text and graphics all displaying properly and cleanly? As the project develops, you also have to trust that the big picture is still good, so you can start to focus on smaller details. 

I find it much easier to break things up, so each time I’m looking at the material, I’m focusing on something different. Each run-through groups similar things together: 

  • Consistency: I’ll start by comparing something to the previous version. 
  • Functionality: Is the format and functionality acting as it should? Does it let me get through the course as intended?
  • User interface: Does all the navigation work as intended? If there’s gating through the program, does that work properly?
  • Language: I’ll run through a developed module and listen to the audio, focusing on the script. Is everything covered? Visually, I’ll look to see if there are grammar issues or typos in the text on screen.
  • Audio: Then I’ll listen to the language. Do things sound right? Was everything said? 
  • Visuals: Finally, I’ll focus on the visuals and the timing of animations. Are things aligned correctly? Sized proportionately? Colored properly?

By breaking it up, there’s less that you can miss each time you look at it.

I try to spend more time on the new things. If something has changed from the previous version, then it still needs to be verified as correct, since I haven’t seen it yet. But if something has already been fixed and I’ve seen it fixed, I can spend less time on it, focusing more on what I haven’t seen or focused on yet. This means that each review should take less time, since I’ve already covered some of it before.

Jean: What’s one of the biggest difficulties you encounter in QA’ing Illumina programs?

Ashley: General or non-specific review feedback tends to slow things down. For example, we’ll receive a comment from a reviewer that a certain aspect of a program isn’t working right, with little description about the issue or other specifics. When one reviewer encounters a functional issue with the program that other reviewers don’t encounter, the developer and I need to recreate it in order to debug and correct it.

In one recent project, a client reported that one of their reviewers said that “after a few screens the program stops working”. It took multiple rounds of communication to determine exactly what the reviewer was seeing, and then a lot of digging to pinpoint exactly how the program was failing, and the specific browser and system combination where the error was occurring. After searching community forums to match the description of the bug with the system specifications, we found that the error had to do with a recent browser update and settings configuration. QA was there to help the developer recreate and identify the bug, but as there are so many possible platforms and configurations to test, that one was a real puzzler! We lost some time but eventually solved the problem.

Jean: What’s your favorite part of QA?

Ashley: As you can probably tell, I like solving puzzles and I pay attention to patterns. We have some clients who require titles in title case, and others who require sentence case. Some who want all bullets to be complete sentences, prompts in specific colors and specific phrasing. US or UK English…  and so on. We need to align with our clients’ corporate standards and those are all decisions that need to be documented in a style guide. I like diving into a course and making sure everything matches with the established conventions. It makes everything look clean and uniform, so it’s easier for learners to get used to one style of something. Knowing what to look for before I start is nice, and I like learning every client’s unique styles.

We’ve been using a number of tools that have made the QA and client review process much more streamlined and efficient for all involved: “Review My eLearning” and “Articulate Review”. I can’t say enough about them. They let reviewers pinpoint exactly where an issue is occurring and “force” reviewers to document what the problem is, and make it easier to address. There has been a little reluctance on the part of some of our clients to use these tools but those that do use them for their reviews have been very pleased.

Jean: What advice can you share for clients and practitioners to avoid some of the common errors you contend with?

Ashley: For other QA practitioners, focus on a few things at a time so you’re not overwhelmed. It’s okay to look at something three times if you’re looking at something different each time. Many people want to optimize the learning and fill as much information at a time as they can. So it’s okay to just focus on visuals, figures and graphics first to make sure it’s correct. Then you can come back to listen to the audio and compare it to the storyboard or script, since the information is presented differently. All the elements come together to form the big picture, but less gets missed when you work on it section by section, piece by piece.

For clients, if you notice an issue, or an item that needs to be addressed, be as specific as you can in noting when and where you saw it. Complete details are helpful and save everyone time. Our goal is for each review – storyboard drafts, Alpha version, Beta version, Final version – to identify less and less issues, so make sure your review is as thorough as possible. Make sure also to look at the comments of other reviewers on your team to make sure that the solutions to addressing the issues are consistent. Like the saying “a stitch in time saves nine”, sometimes when we have to address an issue or make a change that could have been identified earlier in the process, we need to take the equivalent of “nine stitches” instead of one.

Thimble and stitch

Jean: I like that metaphor a lot. When we build e-learning with our clients, we’re working together to quilt your learning into a masterpiece. Piece by piece the puzzle comes together, with elements from different sources. We need to check each piece as things are assembled so there isn’t a lot of rework in the end. As the QA Specialist, you ensure the overall design is consistent, and the stitching creates a perfect pattern. Thanks for taking the time to chat with me, Ashley. I’m not a details person, so I always appreciate the work you do, catching the things I don’t even see, to ensure Illumina’s quality output. Thanks for what you do, and for being an integral part of our team!

Illumina on the Move!

We’ll be presenting at some upcoming conferences. Track us down and say hello if you’ll be there.

  • September 25-27: Training Magazine’s Online Learning Conference in New Orleans: On Wednesday, 9/27, Jean will be presenting UX/UI: Applications for Instructional Design and part of the Learning Leaders CoffeeTalk morning panel.
  • February 12-14, 2018: Training Magazine Conference in Atlanta: On 2/13, Michael and Jean will be presenting Rubrics 101: Improving Participant Performance
  • February 27, 2018: Training Magazine Online!: On 2/27, Michael and Jean will be presenting an online class on Working Out Loud: Lessons in Process Improvement and Collaboration, sharing our discoveries of the last year.

Interested in attending the Training Magazine Conference? Use our discount code TSP2 to save $250 on your registration!

Are your elearning results less than what you’d like them to be? Drop us a line, or give us a call, and let’s chat about it. Let our award-winning team help you create Good elearning. Done well.