Path of Exile 2 has "never experienced this type of failure before"

In the wake of a major outage caused by a Path of Exile 2 update, developer Grinding Gear Games apologizes for the disruption, explains what went wrong, and says it's taking measures to prevent a repeat of the issue. What began as a relatively simple crash bug quickly escalated into 70 minutes of downtime and server lag for a lot of players, putting a dampener on the first week of the action RPG's new season. Fortunately, GGG was able to recover things in relatively good time, and says it has now pinpointed the cause.

It started with an update early on June 4 (in New Zealand, where the Path of Exile 2 studio is based). Just nine minutes after a hotfix patch, the team began investigating reports of many users being disconnected, running into crashes, or suffering poor server performance. "It was unclear at this time how many players were affected," GGG notes, but says that within an additional 15 minutes it made the decision to roll back its hotfix. Unfortunately, it didn't work.

This was due to the sheer volume of 'crash dump' files being created, which were so numerous that it filled up all empty space on the servers, causing their performance to crumble and blocking the hotfix from being deployed. These files are used to identify potential problems and debug, but can be quite large as a result, as they contain the full memory of the process at the time it crashed.

"We have never experienced this type of failure before, and were totally unprepared for it," Grinding Gear Games remarks. It was 15 minutes after the rollback attempt before it realized that the crash dump files were preventing its completion, and a further half an hour before the backend engineers were able to restore the servers to a stable state.

In total, the incident lasted for an hour and ten minutes, during which time many PoE 2 players were either unable to log on or faced severe performance issues. "We apologize for the disruption caused, and will try to avoid this type of situation in the future," GGG says. It is implementing a failsafe to stop crash dumps from filling up entire server disks, and will take advance measures for future updates to improve the speed of a rollback should one be needed.

What actually caused the problem in the first place? A missing data file connected to the passive skill tree. "Due to a slight difference in our packaging system between staging and production builds, this file was incorrectly not included in our production hotfix package." Anyone who was using a radius Jewel covering the area on the passive tree where new nodes had been added would therefore cause the server to crash.

It's an unfortunate bump in what's been a strong launch for the ARPG's new season, and its last before the planned 1.0 launch at the end of 2026. While the endgame overhaul and streamlined campaign have been a big success, there has been a larger than usual bug count. To GGG's credit, it ironed a large majority of these out within the first week, and even overcame this particular hurdle within a reasonable time.