These are the slides from the my GDC 2016 presentation with Heather Chandler, Senior Producer on Fortnite (Epic Games). Therefore, this presentation is about both UX’ (Celia) and Prod’s (Heather) perspectives. You can watch the video of this presentation here. Also, if you’re interested in UX, go check our Game UX Summit happening on May 12th 2016 (we have an amazing lineup! :)).
UX Perpective (Celia) – The influence of User Experience is growing in the gaming industry. UX professionals – who are now well established in industrial design and many tech companies – are eager to start working with the video game industry.
Prod Perspective (Heather) – However, many teams see UX as a threat – weird aliens are invading and making a mess. Why did we need them again? We were doing fine without them (or so we thought). This is kind of what happened at Epic and the purpose of this talk is to provide both perspectives, so there is better understanding between development and ux. If we understand each other we get more out of the relationship. The relationship got off to a rocky start – the UX team was pushing to do a lot of testing early in the process, so they could acquire useful insights; but the production team thought there were bigger challenges to deal with – they already knew there were UX issues, but were planning to fix them later.
Before we start, here’s a quick explanation of how Epic Games is structured: we have multiple product teams as well as support/operation teams since Epic is self-published.
There are UX designers within product teams, but the UX team provides extra help on UX design if needed and provides the tools, methodology, and knowledge to conduct UX analyses and UX research. The UX team is supporting all of these product teams.
Fortnite was the pioneering team that first started to work with UX. Fortnite is currently live in its Alpha stage – it’s an action-building game with RPG mechanics.
Let’s start by discussing misconceptions.
In many ways, UX is a hedgehog that wants to hug the dev team and be loved and helpful. From the dev team perspective though, the spikes aren’t appealing and this new creature is suspicious. But it’s all a matter a perspective. Let’s start from the UX perspective.
Celia – Here’s a quick definition of User Experience: it entails what it is like for the targeted user to interact with a software, including how engaging the experience is (cf. Isbister and Schaffer, 2008), relative to the design intentions. UX is mainly about making sure the design and the business intents are the ones ultimately experienced by the target audience of a product, system, or service. It uses neuroscience and psychology knowledge and applies game user research methodologies (e.g. playtests and analytics) to make sure that the game has good usability and is immersive.
UX has its roots in Human Factors and Human-Computer Interaction (HCI). It’s about looking at how a user understands and interacts with a system without the guidance of the humans who designed it. So it’s about aligning the developers’ mental model with the player’s (cf. Norman, 1988).
NOTE! When we say UX here, we talk about the whole UX initiative: UX design, UX testing (user research), UX strategy, etc. There are a lot of misunderstanding about these different UX disciplines, but this is for another talk 😉
Celia – When I started in the entertainment industry 10 years ago, UX was not really a thing yet. My title wasn’t “UX something”. The first time I had UX in my title was at LucasArts, so only about 4 years ago. And it wasn’t even my official title. Every time I start working with a new team, I face the same misconceptions about UX that I first need to debunk. It’s like my Groundhog Day of UX.
Here’s my top 5:
Heather – UX is going to hamper the designer’s creativity and make the game TOO EASY for the players! You’re going kill games like Dark Souls.
Celia – We would never dare do such a thing! In reality, the main purpose of UX practices is to offer the experience the designers have intended to their target audience. Therefore, if your audience is hardcore gamers and the experience you want for them is suffering, then UX guidelines will absolutely help you accomplish your sadistic goal.
Heather – All this is just common sense – we don’t need to test the UX, the player will definitely understand what they are supposed to do!
Celia – There is some common sense for sure, although issues are always easier to spot after the facts (hindsight effect). Not all UX recommendations are common sense though. The human brain is filled with perception, cognitive, and social biases that affect both the developers and the players. It’s for a reason that researchers from any field use very standardized protocols to test their hypotheses; and that’s because it’s very easy to miss or misinterpret what’s going on.
Perception is a construction of the brain. UX will help you figure out faster and more precisely what the real problems are.
Heather – UX is just another opinion on our game, the opinions you provide are no more or less meaningful then someone from marketing.
Celia – Game developers have to deal with many opinions for sure; from within the game team, from marketing team, publishing team, executive team, etc. You can therefore perceive UX feedback as yet another opinion you have to deal with. How annoying! However, UX processes are meant to test hypotheses through rigorous research and anticipate problems through analysis. There, in reality, UX experts do not give opinions, we provide an analysis based on their knowledge of the brain, past experience, and data when it’s available.
Heather – We don’t have enough time to do UX – we have to make the actual game! Our money is better spent elsewhere, on art and programming.
Celia – It’s funny that you guys don’t say that about QA testing. Clearly, you invest time and money in ensuring the game will ship without bugs. So how can shipping a game with UX issues be less of a problem? If your game ships with critical UX issues, it’s gonna affect your sales and everyone will be sad: the dev team because the game is not achieving what they have hoped for, the executives because it doesn’t make money, the players because they are frustrated with the game… The UX process is an investment: Don’t ask yourself if you can afford thinking about UX, ask yourself if you can afford not to.
Heather – Okay, that’s a good point. So let’s, “UX the game” later, when WE’RE ready.
Celia – You know what’s gonna happen with that mindset? The team will iterate within a closed circle and we’re gonna look at you from afar knowing we could have helped you earlier in the iterative process. At some point, we won’t be able to refrain from it: we’re gonna waive at you asking to start testing and collaborating. Usually, the answer we receive is that’s you’re not ready, the game looks ugly, it’s broken, etc.
The danger is that the game ends up being tested and evaluated too late, when only quick patches can be done before the game ships. So, I’m awfully sorry but we cannot “UX this” at some point after the architecture and important decisions about the game are already made. By the way, “UX” is not even a verb…
Heather – Ok, ok I get it. But consider my perspective… this is what I have to deal with.
Celia – The developers don’t really understand all the subtleties of UX.
Heather – While they may not have a PhD in UX, a designer understands that providing an easy way for the player to interact with the game is part of good game design. The dev team may not want to listen to UX feedback because, well, you are not designers. However, most of the time the dev team is just busy putting everything together, and then they free up some mental space to listen to UX feedback.
Celia – The dev team doesn’t want to work with us – you think we are a nuisance!
Heather – I wouldn’t say nuisance… Most dev teams are happy to utilize resources that will ultimately make the game better for the players. If a process is in place that works well for both, the dev team is happy to work with UX as a partner. The real issue is that you may offer feedback that is too detailed and not applicable to where things are in the development process. We need to be aligned on what feedback is useful at any given time in the development cycle. Also, if UX is not something the dev team is used to working with, time is needed to establish a working relationship.
Celia – The dev team always gives the excuse that they don’t have time to do UX testing. This doesn’t require a lot of time on your part – we can handle all the tests and reports!
Heather – I will admit that in the end, a good UX process saves time for the dev team. But initially, adding UX into the process requires more time. You have to allot for tests, reviewing and tracking feedback, educating the team on UX practices, and so on. It needs to be integrated into schedule, otherwise it will be ignored. It’s important to note that integrating UX into the development process can’t be done overnight and just shoe-horned the process. You need to prep the team, get buy-in, and come up with ways that gradually introduce it. Just having tests and generating reports is not actually integrating UX into our process.
Celia – Implementing UX feedback is not that hard! It’s just a matter of priority, right?
Heather – Your team can provide great suggestions on how to make improvements to the game. These findings and suggestions may seem simple to implement, but in reality I have to treat this as another form of feedback in the development process and weigh it against all the other priorities. Rewriting text or changing colors may seem easy, but they still need to be tasked out, completed, and tested. Larger changes require more people to be involved, more testing, more iteration, etc. UX changes can be a big deal to implement, especially if planning has already been completed.
Heather – Ultimately, the UX team is another input that production has to account for. The key is to determine which inputs are going to have the most value for the game team at whatever point they are in the process. The dev team also needs to understand how to prioritize the UX feedback against things that Execs and Marketing want.
Since UX is based on data, it is easier to determine when a piece of feedback is useful to implement. And once it is implemented, it can be retested to see if it had the desired impact. So, it does make sense for us to work together throughout the development process to get the most from the relationship.
The most important thing your team can do is to gain our trust, and vice versa.
Heather – Now, that I’ve said my piece and represented for the dev team, Celia is going to discuss the shift that occurred when she started to build bridges with the them. She had to face this alone for the first year, since she didn’t have me as her partner in crime. She laid a lot of ground work for building this relationship.
Celia – Well, I’d say that you first need to overcome these misconceptions, and I would argue that the first step has to be done by the UX peeps, because *we* are the new partners, the aliens. I had to stop bossing the team around about usability guidelines and stop pestering each time obvious UX rules were being violated during development.
Instead, I tried to “UX the UX”: I listened to the team (my audience), tried to understand what they needed (the intended experience for my audience), and I used their perspective to accomplish what needed to be done. This part describes this journey.
Celia – The first usability tests and reviews that I did for Fortnite (yeah I was sort of a UX team of one) were seen as being bossy when the team already knew that things were not ready and not working well. When I just wanted to get the ball rolling and start a pipeline, the process could be seen as just stating the obvious and some devs thought they had to deal with me as an “authority” that would report back to execs what was going wrong. UX was trying too hard to put in place a process before proving its value for the dev team. Because I was initially too zealous, I wanted to test everything anytime to prove the value of UX. I started to be felt as the “usability police”. Believe me, you don’t want to become too annoying…
Celia – At that time, with the help of a few UX advocates, I looked at where the project was, asked questions to designers to understand their intentions, and to executives to understand the business plan. Then, a UX test was conducted, or a UX evaluation was done. UX provided a huge report of all the issues, just to have a heartbeat, hoping to start a conversation and get feedback from designers to understand what was by design, what was already in the pipeline to fix, what was new UX issue to them, etc. However, the report was HUGE.
The problem is that the report was sent from a team-of-one they didn’t know well, and I never really got a chance to discuss with the devs, except for a few designers interested in UX. So in the end, the dev team was not particularly excited by the whole thing, connection was not made, and I did not have the full picture of the dev’s team plan.
Celia – So I teamed up with the designers that were interested in UX, and instead of doing a big review of the whole experience, we chatted about their main issues and challenges. That was the starting point of a more tailored UX-Dev relationship. At that time, there was no UX designer on the team so I helped them with some UX design challenges (such as making dirty mocks to help them communicate their vision with the Execs). We also started to test smaller things (icons, HUD, core loop, etc.) with very specific design questions so designers were taking part in the UX process.
The picture on the left is my quick and dirty illustration of the designers’ vision of the metagame from 3 years ago (my art :p) to what Fortnite’s HomeBase is today. I’m obviously not a UX designer but I just used basic Powerpoint functionalities and threw stuff together to help everyone visualize the design direction.
Celia – When Epic was ready to build a UX lab (fairly fast which was great), a lot of people wanted that we streamed the tests so the devs could watch them from their desks.
I refused. I wanted the UX secret room to become a place where not only we could physically be behind the players, but also a place where we could joke and exchange ideas. Watching a playtest then became a commitment (not something the devs could do while multitasking) and a social experience (with other dev team members and UX team members).
Therefore, the secret room is completely sound proofed and has a cool ambiance: it became a place where we could mingle and even party (once a month I organize a party in the ux lab – because why not). This is how the ux tests became a fun aspect of the process. Sort of. Yes, it’s hard to watch players not getting your intended design, but doing so in a casual and friendly environment makes it much better.
Of course, you might not be able to build such a fancy lab, but there are ways to do it cheaply: just a room where you put playtest participants and stream the capture to another room. The most important thing is that the devs should not be the ones conducting the tests. If you cannot hire a UX researcher, try an intern in experimental psychology or Human Factors to run your studies, as it’s quite difficult to remove biases if you do it yourself. If you can’t have someone else do it, make sure to ask a Games User Research person some advice about how to run the tests. They are out there, on LinkedIn and Twitter, eager to help.
Heather – I started working at Epic during the shift. I understood the value of UX based on previous games I’d worked on and I was excited that Epic had an entire UX department! I wanted to understand why the dev team was uncomfortable with UX and figure out how we could improve the process. When the game was in early development the devs didn’t see the value of UX feedback. The team knew that game was very rough, and they were experimenting with various features. UX was seen to be something to do only when the designer had something they felt was ready for testing, and it was treated as another form of feedback.
Celia – They didn’t fully recognize that UX feedback is more neutral and scientific (when done correctly). UX feedback is different because UX professionals are trained in cognitive/experimental psychology and Human Factors to understand HCI issues and fix them.
Heather – The team was also overwhelmed at the amount of information provided. There were pages of reports and all the feedback was also tracked in a humongous spreadsheet. The dev team felt like there wasn’t enough time to thoroughly digest the feedback and determine which should be applied to the game (and at what point it should be applied).
Celia – Yeah, UX reports tend to be overwhelming for sure. But we wanna be thorough! Maybe too much. Less is more, even in the UX world, so targeted tests are certainly more actionable and less overwhelming.
Heather – Because the dev team was working quickly to finish milestones, UX feedback was not a priority to review. The reports would be generated, but might not be reviewed in depth for some time. By the time the feedback was reviewed, the team didn’t find it as useful since so many things had changed in the game since the UX test.
Celia – UX feedback should have priority over all other feedback because (if done well) it’s the most neutral and “scientific”. It will help the team take a step back and see their game through a different – less biased – perspective.
Heather – Instead of approaching UX as a new process/system to shoe horn into the development pipeline, we started thinking about UX as a member of the team. We needed them to feel part of the team so they know the design intentions and can adjust their feedback depending on the current challenges of the game.
Celia – Ideally, you want UX to be close to the team while trying to keep a certain distance with the game itself, or else we too will become too close to the game and we’ll start being biased.
Heather – To get the team more used to the idea of UX, we had to get to know each other. Since the dev team was less interested in strengthening this relationship, the UX team found ways to work with the dev team on their terms. UX made themselves readily available as a resource to the team. There was low friction for someone to talk with Celia’s team and get feedback. Anyone from team is free to use them – no need to go through any hoops. The UX room is centrally located for everyone and the door is always open. Production evangelized UX and made their work more visible to the team (simple things – like reminding people in meetings to consult with the lab, inviting people to view UX tests, and inviting Celia to team meetings). It was important to build on what Celia had established and show the devs how UX could be a part of the team, without becoming another “gate” the game had to pass through.
Celia – Instead of trying to force UX process on the whole company or game team, I first listened to the dev team and partnered with those who were interested in UX. I chose small projects that were less under the spotlight. It allowed us to go through the design – test – iterate – retest very fast so we could make quick wins. These wins could then be acknowledged by other teams and UX love started to spread. By the way, this illustrates how a UX process can easily benefit smaller teams and studios as starting small is actually an advantage. Gradually, after doing a few UX tests and seeing some good results, the team came to appreciate how UX can help them and UX became a concern of everyone, not just the UX team of one. The UX team became a resource anyone could use and a UX designer was hired on the Fortnite team (Derek Diaz). We started planning UX as part of our development pipeline.
Heather – When I started working with Celia, she tracked all the UX feedback in this spreadsheet. At the time, this was the most effective way. As you can see it’s quite dense. It tracks the UX feedback given to the team, the priority, when it was implemented, and if it was retested. The sheet was used as the main transfer of information between the UX and game team, but was mainly a tool for UX since the devs were not involved in driving or maintaining this process.
This worked ok at the beginning. There were some easy fixes to make in the game. But after a few months, it was tracking over 200 items, and very few of them had been addressed – or had been marked as something to do for the future.
Heather – So when I started working at Epic, I saw this as a one-side conversation with UX doing all the work. The process was not well-defined or consistent, so it gave the impression that UX was something that happened randomly. The dev team was willing to review this sheet with UX on a periodic basis, but it was not an integral part of the development process. It had no clear owner on the dev team side and thus it was difficult to get changes into the game and communicated back to UX for retesting.
And it was Overwhelming! It was very difficult for someone on the team to review this independently and parse through the information and determine what was important or useful to consider for the next round of planning. So they tended to ignore it.
Heather – When Celia and I started working together, she wasn’t alone in defining and driving the process. We discussed ways we could work together to improve what we had and to make UX a more integrated part of the development process. In order to make the relationship work best, we wanted our process to work in sync with our releases. On Fortnite, we began by releasing online tests every 6 – 8 weeks and have been reducing the time between releases. We work closely with UX to determine what is going to bring benefit to new content released in our Online Tests.
Heather – We started having weekly one on one meetings to revamp the process and make it more valuable for the team. We identified areas where the process needed to change. One of the first things we did was convert all the existing UX information from the Spreadsheet into JIRA. This immediately made it more accessible to the team and gave UX more visibility within the priority stack. This is a screenshot of the UX dashboard, which we use to see what needs to be implemented, triaged, and re-tested.
Heather – We also implemented a few other things to increase the touchpoints between the two teams so we were more closely aligned on goals and priorities. We looked at the major milestones we were releasing in the next 6 months and defined the priorities of the features to test. We wanted to closely align the UX priorities with the game priorities, so that there was value in what was being tested. We also defined the UX testing goals for key milestones. We formalized at what points testing occurred in development and what type of testing would occur (3 day UX test vs. 1 day UX test).
The UX tests were part of the development schedule and publicized to the team. The teams were now aware that they had to plan for UX tests at these points – a step in the right direction! We made concerted efforts to invite the team to the UX tests (it’s really valuable to watch other people play your game), and made sure they used UX as a resource early and often.
Heather – This is the current process we use for the UX/Dev pipeline on Fortnite. Since we are a live game, this process covers one full release cycle that includes planning, building features, testing, implementing feedback, and retesting. We are closely aligned and UX support is more integral in our development process.
Here are the details:
Celia – It begins when the team specifies their hypotheses for the next online test. Questions come in from Devs, Analytics team, Marketing, or Execs. We determine what questions need to be answered (it’s important to always have a purpose when planning a test). Once the questions are defined, the UX team prepares accordingly an experimental protocol. We also adjust the test with the scope. Sometimes we test the live build that has just been pushed in long sessions, but we also do shorter and quick-and-dirty tests with prototypes (on paper, interactive prototypes, and even sometimes directly in the editor).
Celia – Given the focus of the test and its protocol, we plan when/how/with whom to do it and we decide which build to use.
Heather – Once we have the testing requirements outlined, I work with the team to plan for which build to use. We need to make sure the features we want to test are working. If the build has blockers, the test is cancelled. So we review the build two days before to determine if it ready or not. If not, the test is canceled or, more likely, re-scoped. If the team knows a few weeks ahead of time about the test, it is much easier for them to prioritize their tasks and bugs in order to get a stable build.
Celia – UX tests are conducted and the team observes the participants in real time. We invite the team to come watch.
Heather – And I try not to schedule meetings on the days we have tests so the team is freed up to watch them.
Celia – Once the test is over, we analyze the data and a detailed UX report is generated. But more focused this time.
Celia – After we send the report, we plan a quick meeting to go through the report together, so we can make sure we’re aligned and the team can give us their feedback directly regarding the issues raised.
For each issue, we decide together if it’s by design so we let it be, if it needs a callout so the UX team enters a JIRA, if it’s already in the process of being fixed so we follow-up to retest the feature later, etc. When UX-lab enters JIRAs, we enter detailed info: description of what we observed in the test and/or description of a violated usability heuristic (e.g. consistency violation, or the form of this item conveys a different perception than its functionality, etc.). We add screenshots and videos from the UX test. We tag it “UXLabFeedback” so all issues entered with a UX concern appear in a dashboard. Before every new UX test, we consult the dashboard and look at what issues were marked as fixed so we can verify if the UX issue is indeed resolved.
Heather – Once UX enters JIRAs, the triage process is a way for the dev team to review them and determine if the issues will be addressed in the next release, a future release, or not at all. If the team disagrees with any of the UX feedback or wants to change the priority, they will meet with the UX lab to discuss the options. Once both groups are in consensus, the task is prioritized appropriately in the development schedule.
As you can imagine, something that the UX team marks as a Pri-1, may be viewed as a lower priority for the team. Some of the feedback may require large reworks that can’t be addressed immediately and thus will have to be part of future planning. Some of the feedback may be in opposition of the design intent and require a longer conversation with UX about how to solve the problem. The strike team leads are responsible for reviewing the feedback and initiating further conversations with UX about any questions or disagreements.
Heather – Once a UX task is completed, we marked closed in JIRA and so that Celia’s team knows to re-check it in the next test.
Celia – The data is crossed-checked with information from other sources (Analytics, Customer Support, Marketing, UX surveys sent to the alpha participants etc.) and communicated back to the team so they can come up with a plan for improving the UX.
Then, we communicate to the executives and Publishing team the relevant information and the dev team can already expose their plan to fix the issues raised. It’s a collaborative process; we never give feedback directly to the publishing/executive team without first syncing with the team dev. There are no surprises and certainly no back stabbing. We’re here to support the dev team accomplishing their goals, not to act like self-absorbed smartasses.
Also, since the UX team is neutral, it doesn’t serve any agenda other than fighting for the user. So it helps everyone in the room to re-adjust their personal perspective and opinion on the game (ok, this is not always successful I have to admit).
Lastly, it’s very important that the UX team emphasizes not only the issues the game has, but also all the progress that the dev team has made so far and the UX issues solved. Execs love to see things progress (and love when issues are pointed out so they are reassured that they are taken care of). The dev team loves when we acknowledge how much progress was made (backed up with science, even!), so everyone is happy. Most of the time -ish … (developing games is after all a hard and an emotional endeavor for everyone).
Heather – While we have a unified and defined process it still needs work! The first part is working well, now we need to focus on improving the triage and verification phases. The producers need to ensure that JIRA is being updated and team is actively tackling UX issues (and not just marking them for future milestones). If things are not filtered and assigned appropriately, they lose visibility. Both teams need to have a better understanding of when UX feedback is implemented so that it can be re-tested. It’s not truly fixed until it’s retested and shows that issue has been addressed. In order to get and maintain high visibility of UX, the dev team members are expected to observe UX tests. And we always need to be open to adjust our workflow and process to make things better!
As you can imagine, there is still work to be done, but we’ve come a long way! We’ve learned a lot about each other and are less afraid. It’s all a matter of perspective.
And now, UX is truly part of the process …
Heather – Debunk misconceptions. Don’t be afraid of each other!
Celia – Start small with developers interested in UX to demonstrate UX quick wins
Celia – Don’t be UX police, instead work together to be successful and measure/communicate the progress
Heather – Establish a strong feedback and implementation loop and make sure both teams understand how the process works. Adjust as necessary.
Both – Celebrate together!
