PERMA-RUN: Research Phase

Preface

I started my research with a literature review to identify the existing research gaps in the domain of cybersecurity. It was evident that only a handful number of games were available for smartphone security awareness which covered only some of the security use-cases. Thus, I designed and developed PERMA-RUN, a persuasive game to improve user awareness about smartphone security. I tried to cover most of the smartphone security scenarios with the game. After the game development, I conducted a heuristic analysis with the help of other researchers to avoid personal bias and inform my design decisions. Later, I conducted an evaluative research study to compare the game's efficacy with an existing intervention. Currently, I am experimenting with tailoring the game according to the player's motivational orientation.

Heuristic Evaluation for Playability

  • Uncovering design issues at early stages - Heuristic Analysis helps to uncover any design issues during the early stages of development, thus ensuring that the basics are in place before testing the game with the players. After the game's first build, I conducted a Heuristic Evaluation for Playability (HEP) with the help of seven researchers to evaluate the following core heuristics - gameplay, game story, game mechanics, and game usability. I used a survey to evaluate the heuristics and perceived persuasiveness of the game. I also collected open-ended feedback about the game. The researchers played the game for about a week before filling up the survey.

  • I used descriptive statistics to analyze the survey data and visualized the mean heuristic and the perceived persuasiveness scores. In tandem, I conducted thematic analysis on the qualitative data using the affinity diagram method. I utilized the survey results and the results from thematic analysis to generate and prioritize the actionable insights in the form of a usability report (Click Here to view the Usability report).

Evaluative Research-1

After implementing the design changes according to the results of the Heuristic Analysis, I wanted to test if the game improved users' awareness and their intention to follow safe smartphone practices. I conducted a literature review to see how to persuade users to follow security practices and how their behaviour intention is measured. I found that existing researches adapted Protection Motivation Theory to measure users' intention to follow safe practices by measuring various behavioural constructs.

Evaluative Research-1: PERMA-RUN vs Educational Document:

  • Measuring smartphone security awareness - I adapted the Protection Motivation Theory (PMT) survey to measure users' intention of following safe smartphone security practices and check if it changes over time after playing the game. I conducted a 2x2 Between-Subject study to compare the game's efficacy with the best practices educational document published by NSA to see which intervention performs better in terms of improving smartphone security awareness.

  • The study followed the classic pre-post design where the user fills a survey (for measuring the behavioural constructs) before and after playing the game. Later, I also conducted a semi-structured interview with the users from both groups to gather feedback about their experience with the intervention.

  • Analysis Results - The game performed better for most of the behavioural constructs measured using the PMT scale with a 72% increase in self-efficacy (14% for the document) and a 78% decrease in response cost (32% for the document). From the thematic analysis of the qualitative data, it was evident that the game was motivating and interactive.

  • Users who read the document were overwhelmed by the sheer amount of information provided by the document and wanted an interactive and easy version. Users liked the retro-themed game that reminded them of their childhood games. Some of them tried the security tips from the game and related them to their own experiences.

Data Visualisation

Heuristic Analysis

  • View

Frequency Analysis

  • View

Frequency Analysis

  • View

Quiz Analysis

  • View

In-game Log data analysis

  • In-game Data Collection - Apart from the data collected via surveys and the interview, I also collected in-game log data/telemetry data during gameplay. I logged the in-game coordinates of the places where players lost lives along with the in-game quiz answer attempts.

  • I used the in-game coordinate data to generate a frequency graph to check the places in-game where players lost lives to gauge the game's difficulty and adjust the level design accordingly. I created a frequency chart using the in-game quiz data with which I was able to estimate the security concepts that the users were familiar with.

Evaluative Research-2

Existing research shows that users are motivated towards their goals either by gains or loses. I tailored the game according to users' motivational orientation using regulatory focus theory (RFT). Some people might be motivated towards an objective/goal due to the gains they might get out of it. Some people might feel motivated to stay safe/avoid losses or adhere to social norms. After a brief literature review, I learnt about Regulatory Focus Theory, a goal pursuit theory to tailor persuasive messages according to a user's motivational orientation.

Evaluative Research-2: Promotion Focus vs Prevention Focus

  • Tailoring Game - I conducted a 2x2 Between-Subject study with 300+ participants to compare two versions of the game tailored according to Regulatory Focus Theory.

  • I have tailored the in-game security suggestions and a few game mechanics according to the Regulatory Focus Theory's two motivational orientations:
    Promotion Focus- The user is motivated by gains or positives.
    Prevention Focus- The user is motivated to stay safe or avoid losses.

  • Study Design - The study is a pre-post design where the players answer survey questions before and after playing the game. I also conducted a semi-structured interview with the players to gather their feedback about the game and performed thematic analysis to analyze the qualitative data. From the results, it was evident that the tailored version of the game performed better than the non-tailored version.

    The detailed results of evaluative research-2 will be updated in the near future. For now, you can download the android game here!

UP NEXT

TREECARE: A Persuasive Gamified Step Tracker