Preamble: this issue is meant to document the findings I've made so far and organize an exploratory effort into improving the situation. It's unlikely that we will be able to entirely fix the problem, due to native code, BC concerns, poor asset organization by FXS (and likely mods which either followed FXS's poor example or just had no guidance at all), combination of these or due to some other reasons. However, I do believe that we are in a position to make a notable difference.
Background knowledge
To understand how we got to the situation today, let's first take a look at what FXS had when they started work on X:EU (which WOTC is eventually based on) - the default UE1-3 asset system. It was designed quite a while ago (Object.uc starts with Copyright 1998) and had a couple of major assumptions:
- Levels (
*.umaps) are the main unit of organization of assets/logic and are the entrypoint to everything that happens
- Game will played directly from a DVD (on both consoles and PCs)
This meant that during gameplay, "hot-loading" assets wasn't feasible, so instead everything that could've been potentially needed in a level was loaded upfront (during a loading screen). This was done by recursively following references between objects (either at runtime or by the cooker) until everything was in memory. When the player went to a different level, everything referenced by the previous level was unloaded and the process was repeated for the new level. There were some other (minor) ways to load assets, but they were either not applicable to content that needs to be loaded from time to time (not always), didn't work on consoles or confused the cooker, so I'm not covering them here.
It is obvious that such approach wasn't feasible for xcom-type game. For example, it would involve loading all the enemies that exist in the game, regardless of which ones appear at the current point of the campaign. Levels also weren't the driving force behind the gameplay - they were simply a collection of semi-static environment. You get the idea.
As such, FXS came up with a more specific system. I don't know how it evolved (and neither that is relevant), but in WOTC it lives mainly inside XComContentManager. The basic functioning of the system is simple:
- Before a layer transition (e.g. strategy -> tactical), a list of
RequiredArchetypes is prepared - unit cosmetics, weapons, you name it
- When the root map of next layer is loaded (
Avenger_Root for strategy, the plot for tactical), the content manager also loads everything in the RequiredArchetypes, during the loading screen
This allows to bypass the "world revolves around levels" logic while keeping the benefit of not needing to keep doing IO during gameplay, Sounds great so far.
In order to prepare the RequiredArchetypes, XComContentManager::RequestContent is called. While the exact logic is unknown (the function is native), one of main points is that it calls XComGameState_BaseObject::RequestResources on all states. That allows units to add selected cosmetics, etc.
The GC Root Set
Before we move to the actual behaviour, let's understand what prevents assets from being unloaded (from memory). Unreal Engine uses a mark-and-sweep Garbage Collector (wikipedia) as its primary mechanism of memory management. This means that a GC pass will first mark all objects that are in use and then sweep through the entire memory and destroy objects that weren't marked.
How is "in use" determined? Easy - it's everything that is referenced by other objects that are "in use"... But this chain needs to begin somewhere. That starting point is the "Root Set" - a bunch of objects that the GC is told to assume to be definitely "in use".
While the exact contents of the root set are irrelevant to the subject hand, a specific part is of interest - the various singletons, such as the Engine and managers, such as the XComContentManager mentioned above.
And now the important part - everything loaded by the content manager (either via the RequiredArchetypes list or by manually calling RequestGameArchetype) will be added to XComContentManager::MapContent. As such, it will never be unloaded by the GC, whether it's in use by anything else or not, since it's considered to be "in use" by the content manager. Note that this list is cleared during the level transitions - the only way to "release" these assets.
The Reality
When a tactical mission is about to be launched, XGStrategy::LaunchTacticalBattle does a bunch of preparation, including calling XComContentManager::RequestContent at the end:
|
//Tell the content manager to build its list of required content based on the game state that we just built and added. |
|
`CONTENT.RequestContent(); |
For this test, I've started a new campaign with only CHL active (de17827) using debug non-cheat start (gatecrasher skipped) and then went on the first mission with a single normal (non-faction) soldier:
But before that, I also made a little change to the code quoted above:
//Tell the content manager to build its list of required content based on the game state that we just built and added.
ContentManager = `CONTENT;
DumpArchetypes("Before request", ContentManager.RequiredArchetypes);
ContentManager.RequestContent();
DumpArchetypes("After request", ContentManager.RequiredArchetypes);
private simulated function DumpArchetypes (string Description, array<string> arrStr)
{
local string str;
`log("======================================== BEGIN",, 'DumpArchetypes');
`log(Description,, 'DumpArchetypes');
`log("",, 'DumpArchetypes');
foreach arrStr(str)
{
`log(str,, 'DumpArchetypes');
}
`log("======================================== END",, 'DumpArchetypes');
}The result: https://gist.github.com/Xymanek/4198a07ba342fca065ed2508fe7a5682
The extras
While the list is obviously long, let's take a look at a particular part:
[0139.14] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Cnv_Torso_M
[0139.14] DumpArchetypes: SldCnvUnderlay_Std_GD.ARC_SldCnvUnderlay_Std_Torsos_A_M
[0139.14] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Torso_Deco_C_M
[0139.14] DumpArchetypes: Head_Skirmisher.ARC_Head_SkirmisherMaleA_Cauc
[0139.14] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Helmet_B_M
[0139.14] DumpArchetypes: Prop_FaceLower_Blank.ARC_Prop_FaceLower_Blank
[0139.14] DumpArchetypes: Hair_FemBlank.ARC_Hair_FemBlank
[0139.14] DumpArchetypes: FacialHair_Blank.ARC_FacialHair_Blank
[0139.14] DumpArchetypes: Prop_FaceUpper_Blank.ARC_Prop_FaceUpper_Blank
[0139.14] DumpArchetypes: SldCnvUnderlay_Std_GD.ARC_SldCnvUnderlay_Std_Arms_A_M
[0139.14] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Arm_Left_A_T1_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Arm_Right_B_T1_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Cnv_Shoulder_Left_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Shoulder_Right_C_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Forearm_Left_A_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Forearm_Right_B_M
[0139.15] DumpArchetypes: GameUnit_Skirmisher.ARC_Skirmisher_Legs_A_M
This looks like body parts (cosmetics) for skirmishers. But wait, what!? This is vanilla game, FL1 with a squad of only a single Soldier - there can't be a skirmisher on the mission! And yes, there is no skirmisher on the mission.
But then, why is the game so interested in loading their cosmetics? Let's take a look a quick look at the barracks:
Oh.
Yes, the game will load all the cosmetics for all the soldiers when loading into tactical - including dead and captured. Slightly good for avenger defense, huge waste of loading time and RAM (remember - this won't be unloaded until the end of the mission) for literally every other mission.
As you progress the campaign, this will get worse since you will get more soldiers, they will be using different cosmetics, gear, etc. Using different cosmetic mods will also exacerbate this issue.
The "Helpful" Cleanup
But then you are gonna ask
So what's the deal? They are loaded already for strategy!
Here's Process Monitor recording after I press the "launch mission" button (ignore the -noSeekFreeLoading, the behaviour is the same when the game is running cooked):
As we can see, it does indeed load the cosmetics (and other stuff) again, despite us just looking at them in-game moments ago. Recall that the XComContentManager::MapContent list is cleared on layer transition - and then a GC pass is forced to cleanup everything that was used by the previous layer.
So the fact that we just had these cosmetics in memory doesn't help prevent useless IO.
The missing stuff
Despite the fact that we just found out that the list of "assets we will need" is insanely bloated, there is another problem - it's missing a lot of stuff that we will actually need.
Here's the XComContentManager::MapContent at the point when the player gains full control over the tactical UI: https://gist.github.com/Xymanek/5ccebc14eae4e06d945776c3b21b9edd (dumping code). In particular, let's compare the ending of the lists:
[0139.65] DumpArchetypes: DLC_60_FX_Archon_King.P_Devastate_Target
[0139.65] DumpArchetypes: DLC_3_Absorption_Field.PJ_Absorption_Field_Discharge
[0139.66] DumpArchetypes: Materials_DLC3.3DUI.AOETile_NA
[0139.66] DumpArchetypes: ======================================== END
[0468.78] AD_DumpMapContent: dlc_60_fx_archon_king.P_Devastate_Target
[0468.78] AD_DumpMapContent: DLC_3_Absorption_Field.PJ_Absorption_Field_Discharge
[0468.78] AD_DumpMapContent: Materials_DLC3.3DUI.AOETile_NA
[0468.78] AD_DumpMapContent: UI_3D.Targeting.M_HitByAOE
[0468.78] AD_DumpMapContent: UI3D_XPack.Materials.Reaper_Class
[0468.78] AD_DumpMapContent: FemaleVoice7_English.Voice_FemaleVoice7_English
[0468.78] AD_DumpMapContent: XPACK_NarrativeMoments.X2_XP_CEN_T_Sabotage_Transmitter_Dropship_Intro_C
[0468.78] AD_DumpMapContent: FX_Destruction_Fracture_Data.DecoFX_Default
[0468.78] AD_DumpMapContent: X2_UnitScars.Scars_BLANK
[0468.78] AD_DumpMapContent: ConvAssaultRifle.Meshes.SM_ConvAssaultRifle_MagA
[0468.78] AD_DumpMapContent: ConvAssaultRifle.Meshes.SM_ConvAssaultRifle_OpticA
[0468.78] AD_DumpMapContent: ConvAssaultRifle.Meshes.SM_ConvAssaultRifle_ReargripA
[0468.78] AD_DumpMapContent: ConvAssaultRifle.Meshes.SM_ConvAssaultRifle_StockA
[0468.78] AD_DumpMapContent: ConvAssaultRifle.Meshes.SM_ConvAssaultRifle_TriggerA
[0468.78] AD_DumpMapContent: ConvAttachments.Meshes.SM_ConvFlashLight
[0468.78] AD_DumpMapContent: GameUnit_AdvCaptain.ARC_GameUnit_AdvCaptainM1_M
[0468.78] AD_DumpMapContent: WP_AssaultRifle_MG.WP_AssaultRifle_MG_Advent
[0468.78] AD_DumpMapContent: GameUnit_AdvTrooper.ARC_GameUnit_AdvTrooperM1_F
[0468.78] AD_DumpMapContent: GameUnit_Sectoid.ARC_GameUnit_Sectoid
[0468.78] AD_DumpMapContent: WP_Sectoid_ArmPistol.WP_SectoidPistol
[0468.78] AD_DumpMapContent: GameUnit_AdvTrooper.ARC_GameUnit_AdvTrooperM1_M
[0468.78] AD_DumpMapContent: ======================================================= END
We can see that the latter has a bit more stuff added at the end. In particular we can see the meshes for attachments (or rather the placeholders when you don't have the attachments) and enemy units/weapons. While the former is quite straightforward to fix, the latter is more complicated. This is due to the fact that when XComContentManager::RequestContent is called by XGStrategy::LaunchTacticalBattle, the states for enemies haven't spawned yet - they will be spawned much later, by X2TacticalGameRuleset::StartStateSpawnAliens, after the RequiredArchetypes list was already loaded.
This "delayed load" of enemy assets is the cause (or at least one of) for the infamous lag on the spawn/drop in cinematic, since that's when the game is forced to play catch-up on asset loading.
Related stuff
Players have reported that the simple act of having lots of voice packs slows the game down a lot, but the mess described above cannot be responsible for it:
|
//Don't load voices as part of this - they are caught when the pawn loads, or the voice previewed |
This is confirmed above - we see FemaleVoice7_English.Voice_FemaleVoice7_English as one of the assets that were loaded in the "late pass".
#178 Could be further worsening the issue later in the campaign.
cc @robojumper - could this be some kind of a byproduct of an unintended interaction with the history archival?
Preamble: this issue is meant to document the findings I've made so far and organize an exploratory effort into improving the situation. It's unlikely that we will be able to entirely fix the problem, due to native code, BC concerns, poor asset organization by FXS (and likely mods which either followed FXS's poor example or just had no guidance at all), combination of these or due to some other reasons. However, I do believe that we are in a position to make a notable difference.
Background knowledge
To understand how we got to the situation today, let's first take a look at what FXS had when they started work on X:EU (which WOTC is eventually based on) - the default UE1-3 asset system. It was designed quite a while ago (
Object.ucstarts withCopyright 1998) and had a couple of major assumptions:*.umaps) are the main unit of organization of assets/logic and are the entrypoint to everything that happensThis meant that during gameplay, "hot-loading" assets wasn't feasible, so instead everything that could've been potentially needed in a level was loaded upfront (during a loading screen). This was done by recursively following references between objects (either at runtime or by the cooker) until everything was in memory. When the player went to a different level, everything referenced by the previous level was unloaded and the process was repeated for the new level. There were some other (minor) ways to load assets, but they were either not applicable to content that needs to be loaded from time to time (not always), didn't work on consoles or confused the cooker, so I'm not covering them here.
It is obvious that such approach wasn't feasible for xcom-type game. For example, it would involve loading all the enemies that exist in the game, regardless of which ones appear at the current point of the campaign. Levels also weren't the driving force behind the gameplay - they were simply a collection of semi-static environment. You get the idea.
As such, FXS came up with a more specific system. I don't know how it evolved (and neither that is relevant), but in WOTC it lives mainly inside
XComContentManager. The basic functioning of the system is simple:RequiredArchetypesis prepared - unit cosmetics, weapons, you name itAvenger_Rootfor strategy, the plot for tactical), the content manager also loads everything in theRequiredArchetypes, during the loading screenThis allows to bypass the "world revolves around levels" logic while keeping the benefit of not needing to keep doing IO during gameplay, Sounds great so far.
In order to prepare the
RequiredArchetypes,XComContentManager::RequestContentis called. While the exact logic is unknown (the function is native), one of main points is that it callsXComGameState_BaseObject::RequestResourceson all states. That allows units to add selected cosmetics, etc.The GC Root Set
Before we move to the actual behaviour, let's understand what prevents assets from being unloaded (from memory). Unreal Engine uses a mark-and-sweep Garbage Collector (wikipedia) as its primary mechanism of memory management. This means that a GC pass will first mark all objects that are in use and then sweep through the entire memory and destroy objects that weren't marked.
How is "in use" determined? Easy - it's everything that is referenced by other objects that are "in use"... But this chain needs to begin somewhere. That starting point is the "Root Set" - a bunch of objects that the GC is told to assume to be definitely "in use".
While the exact contents of the root set are irrelevant to the subject hand, a specific part is of interest - the various singletons, such as the
Engineand managers, such as theXComContentManagermentioned above.And now the important part - everything loaded by the content manager (either via the
RequiredArchetypeslist or by manually callingRequestGameArchetype) will be added toXComContentManager::MapContent. As such, it will never be unloaded by the GC, whether it's in use by anything else or not, since it's considered to be "in use" by the content manager. Note that this list is cleared during the level transitions - the only way to "release" these assets.The Reality
When a tactical mission is about to be launched,
XGStrategy::LaunchTacticalBattledoes a bunch of preparation, including callingXComContentManager::RequestContentat the end:X2WOTCCommunityHighlander/X2WOTCCommunityHighlander/Src/XComGame/Classes/XGStrategy.uc
Lines 549 to 550 in de17827
For this test, I've started a new campaign with only CHL active (de17827) using debug non-cheat start (gatecrasher skipped) and then went on the first mission with a single normal (non-faction) soldier:
But before that, I also made a little change to the code quoted above:
The result: https://gist.github.com/Xymanek/4198a07ba342fca065ed2508fe7a5682
The extras
While the list is obviously long, let's take a look at a particular part:
This looks like body parts (cosmetics) for skirmishers. But wait, what!? This is vanilla game, FL1 with a squad of only a single
Soldier- there can't be a skirmisher on the mission! And yes, there is no skirmisher on the mission.But then, why is the game so interested in loading their cosmetics? Let's take a look a quick look at the barracks:
Oh.
Yes, the game will load all the cosmetics for all the soldiers when loading into tactical - including dead and captured. Slightly good for avenger defense, huge waste of loading time and RAM (remember - this won't be unloaded until the end of the mission) for literally every other mission.
As you progress the campaign, this will get worse since you will get more soldiers, they will be using different cosmetics, gear, etc. Using different cosmetic mods will also exacerbate this issue.
The "Helpful" Cleanup
But then you are gonna ask
Here's Process Monitor recording after I press the "launch mission" button (ignore the
-noSeekFreeLoading, the behaviour is the same when the game is running cooked):As we can see, it does indeed load the cosmetics (and other stuff) again, despite us just looking at them in-game moments ago. Recall that the
XComContentManager::MapContentlist is cleared on layer transition - and then a GC pass is forced to cleanup everything that was used by the previous layer.So the fact that we just had these cosmetics in memory doesn't help prevent useless IO.
The missing stuff
Despite the fact that we just found out that the list of "assets we will need" is insanely bloated, there is another problem - it's missing a lot of stuff that we will actually need.
Here's the
XComContentManager::MapContentat the point when the player gains full control over the tactical UI: https://gist.github.com/Xymanek/5ccebc14eae4e06d945776c3b21b9edd (dumping code). In particular, let's compare the ending of the lists:We can see that the latter has a bit more stuff added at the end. In particular we can see the meshes for attachments (or rather the placeholders when you don't have the attachments) and enemy units/weapons. While the former is quite straightforward to fix, the latter is more complicated. This is due to the fact that when
XComContentManager::RequestContentis called byXGStrategy::LaunchTacticalBattle, the states for enemies haven't spawned yet - they will be spawned much later, byX2TacticalGameRuleset::StartStateSpawnAliens, after theRequiredArchetypeslist was already loaded.This "delayed load" of enemy assets is the cause (or at least one of) for the infamous lag on the spawn/drop in cinematic, since that's when the game is forced to play catch-up on asset loading.
Related stuff
Players have reported that the simple act of having lots of voice packs slows the game down a lot, but the mess described above cannot be responsible for it:
X2WOTCCommunityHighlander/X2WOTCCommunityHighlander/Src/XComGame/Classes/XComGameState_Unit.uc
Line 2921 in de17827
This is confirmed above - we see
FemaleVoice7_English.Voice_FemaleVoice7_Englishas one of the assets that were loaded in the "late pass".#178 Could be further worsening the issue later in the campaign.
cc @robojumper - could this be some kind of a byproduct of an unintended interaction with the history archival?