Back to blogs
As game budgets climb to thrilling new heights, efficient testing has become critical to keeping costs down and hitting development deadlines. Automated game testing is becoming essential, allowing studios to scale QA to handle the demands of modern games.
The developers at Rare leaned into the power of automation during the development of Sea of Thieves, implementing an innovative testing framework that transformed their workflows and helped them deliver an experience that many gamers now treasure. Fortunately for the developer community, software engineer Robert Masella delivered a value-packed talk at GDC explaining the automated game testing tactics used to QA the online open-world masterpiece. It’s essential viewing for any games QA professional and a comprehensive example of how automation can help QA teams deliver better results with fewer resources. We’ve highlighted and explored five of the key methodologies below.
1. Automated playtesting
As the Sea of Thieves team discovered, automated playtesting can be a game-changer. Bots that emulate player behavior allow developers to explore every corner of gameplay, from combat to environmental interactions, with a level of consistency and precision that’s difficult to achieve manually. As Masella noted, the game’s open nature posed particular challenges that made this approach essential:
“There were very few restrictions on what the players could do and when they could do it. The challenge here was all the complexity that added — the way features interacted meant that we’d have to keep checking all those interactions, all those features, making sure they still worked. So, rather than [having a human tester] regularly check that scenario… we could just ask the game to do it itself with an automated test.”
Masella noted that automation also allowed them to address scenario testing quickly across a broad range of possibilities, spotting bugs that couldn’t have been found by traditional means. By building bots capable of exploring every gameplay interaction, Rare was able to simulate real, unpredictable player actions.
“With so many ways for players to engage with the game — from hiding from enemies in unexpected places to interacting with objects in unusual ways — exploratory tests helped us detect and resolve issues that would have been very time-intensive to catch manually.”
While building your own AI-powered bots can be time-consuming, behavioral AI engines like modl.ai provide AI-driven game testing bots that can integrate directly into your development workflows with easy-to-install plugins for Unity and Unreal. Modl.ai’s bots also include exploratory capabilities, making it simple to uncover hard-to-find bugs by mimicking diverse player actions. This allows QA teams to achieve deep coverage, continuously test edge cases, and optimize scarce resources — all without the need for custom development.
2. Continuous integration and delivery (CI/CD)
Early bug detection is crucial, as fixes made later in development or production cost exponentially more than those identified upfront. Continuous testing in a CI/CD pipeline keeps code quality high, catches bugs early, and ensures that each build is stable and ready for further development. Automated testing processes make CI/CD even more efficient.
To that end, the Sea of Thieves team implemented a pre-commit process requiring developers to run automated tests on any code before it could be integrated into the main branch. As Masella explained, this approach ensured that only stable builds advanced in the pipeline:
“You could only submit changes if the build was green. We didn’t want to allow people to continue submitting something if the build was already broken… That would make it very important if the test was failing… Whoever was responsible needed to fix it as soon as they could because it was blocking the whole team from committing any more changes.”
This automated pre-commit testing process kept builds stable and productive for the entire team. “The manual testers always get a good build to test with,” Masella later notes, which allowed the QA team to focus on gameplay experience instead of troubleshooting broken builds.
3. Performance and load testing at scale
In multiplayer games, network performance and load testing are the absolute foundation of the player experience. Rare used automated network testing to verify server-client interactions, especially in Sea of Thieves, where multiplayer gameplay relies heavily on seamless communication. By simulating player interactions across a network, Rare could catch and address performance issues well before launch.
“We really wanted to make sure that we covered [the performance] aspect of the game with our integration tests… We changed the automation framework that was there in Unreal to allow integration tests to pass execution between the server and the client so that we could check both sides of network communication. We could go back and forth, ping-ponging between server and clients.”
These network tests gave the team a comprehensive view of how the game’s multiplayer components performed in real-time, identifying any lag or communication errors between clients. For studios aiming to implement similar performance testing strategies, modl.ai provides a scalable solution for automated load testing, ensuring your multiplayer game performs reliably and delivers a seamless experience to players. Contact us to learn more.
4. Automated regression testing
Automated regression testing saves QA time, reducing the need for human testers to repeat checks and ensuring code stability across iterations. Rare implemented automated tests to re-check existing game features and interactions after each update, which helped prevent recurring bugs from slipping back into the game. Given Sea of Thieves’ constantly evolving codebase, they found that automated regression testing was crucial to maintaining stability.
“This is going to be a constantly evolving game… responding to player feedback… As you’re constantly changing the code base, you’ve got a lot of risk of causing bug regressions or breaking features that you had already implemented. So rather than rely on just manual testing, we decided to make automated [regression] testing a core part of our process.”
By continuously re-running automated tests, Rare could quickly identify any re-emerging issues and prevent regressions from reaching players, helping them keep the game stable even with frequent updates.
5. AI-assisted visual testing
AI-based visual testing analyzes gameplay footage to identify graphical errors such as misplaced assets, rendering issues, and lighting glitches that are often difficult to catch manually. Automated visual testing can dramatically increase graphic check efficiency, helping developers catch subtle visual inconsistencies that could affect player immersion.
Visual consistency was essential for an open-world game like Sea of Thieves. Rare used automated screenshot tests that captured images from different scenes and compared them against “last known good” versions to catch visual discrepancies early.
“We had screenshot tests that looked in practice a lot like integration tests, but at the end of it, they usually took a screenshot where we then compared that screenshot… against the last known good screenshot of that test. If there were any differences, that meant there was a visual error or rendering bug. This approach allowed us to catch subtle visual issues — like assets not loading properly or lighting inconsistencies — that would have been hard to spot manually.”
By employing these automated visual tests, Rare maintained a consistent level of graphical fidelity across their game environments, quickly identifying small but impactful visual issues before they reached players. Modl.ai recently introduced new functionality to its modl:test solution where bots interact with GUI elements within the game and record videos that are automatically analyzed for missing assets and other visual issues. Taking visual testing to the next level, modl.ai’s solution can now analyze screenshots without relying on a last-known-good reference and even automatically create detailed bug reports whenever issues are identified.
Set sail for automated excellence
The Sea of Thieves team’s success with automated testing is an example worth emulating for modern game developers. By incorporating automation across playtesting, performance testing, visual testing, and more, the devs at Rare transformed their workflows and dramatically shortened development cycles.
For game studios looking to streamline testing, modl.ai offers a powerful suite of tools to help including AI-powered testing bots, automated bug reporting/analysis, realistic player bots, and more. If you’re ready to elevate your testing workflows, enhance team efficiency, and accelerate overall production timelines, contact us to learn how you can add modl.ai’s advanced solutions to your team’s toolkit.
