Back to blogs
Modl.ai sat down with games production executive Rob Sandberg to discuss the evolution of QA, automated testing, and a shift-left approach to game development.
Rob Sandberg is a three-decade veteran of the games industry. From working on massive franchises to his role as Senior Director of Production at Product Madness, he’s seen how QA can make or break a game’s development.
It’s the sort of industry experience we here at Modl.ai love to see, so we jumped at the chance to pick Rob’s brain in a one-on-one interview. Our chat spanned the breadth of QA and automation, his thoughts on some of our recent research, and how he sees game production evolving.
Key Takeaways
- Shift QA left in the development process by embedding QA members within development teams to identify issues earlier and improve agility.
- Integrate automation into CI/CD daily QA processes to continuously check your main branch overnight and catch issues early.
- Simulate actual player behavior in automated tests to identify issues casual players may encounter and avoid costly hotfixes in live production.
- Position QA as a strategic priority by establishing clear KPIs and advocating for QA’s value in leadership discussions.
1. Shift QA left in the development process
Rob’s unique experience in production affords him a top-down view of how development pipelines can operate most effectively. In his experience, integrating QA early is an ideal way to minimize downstream issues. Sandberg explains how an outdated waterfall quality control mindset hamstrings game studios and delays value delivery to the player. To improve quality and time to market, developers need to re-think their QA and game delivery approach.
“The smart developers are the ones who shift left— the ones who have the QA team in the development phase. They’re not testing all the features when they get dumped off to them as a functional test. They’re doing development-level testing. You don’t need 10 or 15 testers doing that. You need 1 or 2, and they should be embedded in your team’s Software Development Life Cycle (SDLC) throughout development, not just at the end of a milestone.”
This “shift left” approach, popularized by DevOps, prevents QA from becoming a bottleneck. Combined with automated testing procedures, it can shave hundreds of hours off manual testing. In surveying more than 300 game developers from across the US, we found that 86% of QA professionals already rely on some level of automation to keep their pipelines running. The more developers can take on QA as a shared responsibility, the less likely it is for costly bugs to make it into production.
Take Marvel’s Avengers, for example, which launched an update causing the game to display each player’s IP address publicly. Developer Crystal Dynamics had to scramble to fix the bug and, in the meantime, urge streamers to stop playing the game for fear of doxxing. It was a heavy blow for a game that relied on high player counts and microtransaction sales for sustainability and contributed to its ultimate downfall. They may have avoided the issue if QA shifted left and caught it earlier.
2. Integrate automation into regular QA processes
Throughout our conversation, Sandberg reminds us of one simple fact of game development: There shall be bugs. Automation is not a panacea for the battles inherent in QA, but it can keep a “heartbeat” on a game’s overall health through automated overnight tests, especially regression and smoke tests. Sandberg says this helps to identify bugs before they reach production, flagging problems that arise as you make changes.
“Your automated QA processes, integrated with your CI/CD pipeline, should be set up to check your master branch on every commit. This ensures that all changes are immediately tested, helping to catch and address any issues early in the development cycle. And then, as you introduce new changes, automation will run tests until, eventually, it flags something and says, ‘Oh, wait a minute. This broke over here. You broke something that you didn’t intend to.”
Rob emphasizes that automation allows for continuous testing, which creates a safety net for production. Instead of a human spamming a key 10,000 times, automated testing solutions can tackle the busywork, which makes it easier to prevent bugs from slipping into live production.
Something as simple as automated processes to check code line-by-line can significantly help because even small typos can have dire consequences. Such was the case for Aliens: Colonial Marines, where an errant “a” caused issues with AI teammates and enemies so severe that it essentially broke the game. It was met with low review scores and caused lasting damage to the franchise.
3. Avoid Hotfixes by Simulating Real Player Behavior
QA testers dedicate their careers to breaking games, but that level of repetition can cause issues. Sometimes, Rob explains, QA teams that are especially good at rooting out bugs can stop playing like real players or are focused on new features and do not have the budget for exploratory testing of the rest of the game. Rob offers a solution: Let machines break the game.
“Sometimes, QA knows how to break a game too well. They know how to hunt for and find bugs but aren’t playing like real players. If you can get your automation to play like a player, you’ll find new bugs to fix. Catching the issues real players will encounter early is huge because hotfixes are the scourge of live service games.”
Automated testing bots, like those offered by modl.ai, can simulate actual player behavior. By using AI-driven agents that explore the entire game state that can be reached by real players, we help studios uncover nuanced bugs that QA teams might miss, reducing the need for post-launch hotfixes. This solution not only enhances game quality but also frees up QA teams to focus on high-impact issues.
The data from those automated tests ensures that problems are caught before they can impact players, which helps developers avoid the most expensive bugs. Rob points out that hotfixes, or fixes applied to versions of a game that are live in the wild, are exponentially more expensive than those found during development. They’re the equivalent of open-heart surgery, requiring a new client update and diverting resources from future development.
There’s no better example of the cost of hotfixes than Cyberpunk 2077, CD Projekt RED’s futuristic RPG. The game launched with many game-breaking bugs, ultimately costing the developer $125 million to fix post-release.
4. Advocate for QA as a strategic priority
Finally, Rob reiterates how important it is for development teams to shift QA left. But he also broaches another subject: It’s time for executives to recognize QA’s value and importance. In his experience, QA is often undervalued; it can be the first department subjected to budget scrutiny when costs need to be lowered. Our 2024 survey of US game developers found that 50% of developers don’t believe that QA budgets are keeping up with the feature expectations of modern live service games, and it’s difficult to see the problems as disconnected.
To solve the problem, Rob recommends implementing clear KPIs and simple metrics to help communicate QA’s value to executives and ensure it receives proper resources.
“QA should have a seat at the table. They should report to the studio head and be part of the senior leadership team. They’re kicked down too often. They should have simple KPIs that executives can digest, speak to quality, and frame what they’re doing so they can get the resources they need.”
Rob argues that when teams relegate QA to a back-office role, they lessen its potential to impact the game’s overall quality and user experience. With more visibility, QA can be recognized as a crucial function rather than a line item.
Wrapping up
Rob’s insights remind us of the transformative power QA and automation hold for the future of game development. By shifting QA left, integrating automation, and prioritizing player-centric testing, studios can minimize costly hotfixes and elevate the player experience. Rob’s call for QA to have a strategic seat at the table speaks to the importance of quality as a foundational element of game production.
If you’re ready to bring these principles into your studio, explore how modl.ai can help automate real-player behavior testing and enhance your QA processes. To learn more about Rob and his work, follow his profile on LinkedIn and check out his blog on Substack.
