Evaluating the User Experience: How Instructors Can Track Behavior

Evaluating the User Experience: How Instructors Can Track Behavior

For eLearning courses, evaluation of test scores alone is not enough to determine the learner’s overall experience. In their presentation at the Lectora User Conference, Peter Sorenson, President of Quizzicle, LLC, Bradley Fredrick, Instructional Designer at Correctional Medical Services and Michelle Allen, Senior Manager at Pfizer, introduced three different viewpoints regarding evaluation, and how Lectora allows instructors to track students’ behavior even from a distance.

Sorenson’s main points
Think about education programs as a living entity. It’s able to monitor, observe, support and modify the actions and behaviors of users while engaged in the program. This is the ONLY way to TRULY evaluate the experience.

Necessary challenges for instructional designers to overcome?

Training budgets are shrinking.

Expectations are growing.

Rapid development is a priority.

For corporations?                                                            For students?

ROI difficult to measure                                                  No penalties for bad behavior

Training viewed as cost                                                  Performance is evaluated afterward

False sense of training security                                      No face-to-face interaction

In the classroom, participation, body language and demeanor are some characteristics that are evaluated to determine success. With web-based training, essentially only participation, attendance and test scores are evaluated. What’s missing? Behavioral data.
Fredrick’s main points
Most tracking data is based on who passed, so instructors aren’t fully aware of what the students are experiencing. How does the person feel using: your product, your course, your interaction, your tests, etc.?
“My Flawed Thinking:” passing an online course should be based on the user passing an exam or assessment where the exam or assessment reflects the learning objectives of the course, so you must learn to write good course objectives. The flaw is believing that by passing they learned everything that was intended. You’re not evaluating the experience because they’re not one in the same.
Allen’s main points, from a management, strategic position
Different types of trainees: IDHT (I don’t have time, not going to do it) IDNT (I don’t need training, I’m too good) TIMC (Training is my career, can get out of work to waste time) IATO (I appreciate training opportunities)
Beneficial possibilities of evaluation: Proof of comprehension. Demonstrate level of interest. Identify topic engagement. How employees learn best. Data to analyze.
Sorenson’s solutions
Observe the learner. You must make sure learners are engaged in the topic.
New way to develop training: think about actually using your instructional design knowledge, developing training to acquire information and running reports that might indicate actual comprehension.
Benefits of robust evaluation for instructional designers: Identify trouble spots in a course. Identify what works in a course. Build profiles of successful behaviors

For corporations:                                                                 For students:

Prove ROI                                                                           More personalized experience

Identify outliers                                                                    Real time support

With Lectora courses, you can add script scans to courses to evaluate online “behavior.” It watches browser window events, input events, element interactions, element considerations, page event sentencing, event duration, latency/response time and navigation behavior. The software does the thinking for you by interpreting the individual’s behavior according to his or her actions on the page, then puts it in a numerical or graphical display for you.
Re-evaluate: “Lectora is the best tool on the market to build actual training materials. At the root of it, it allows people…to look down the road and see how we can be better,” Sorenson said. “It gives you the ability to go in and build stuff that can run in the background, and create the wealth of data that you can take a look at, evaluate and review. [Lectora offers] true evaluation for the instructional designer, corporation, individual engaged in the training. And, no one else does it.”