CurveCodecv0.2.0 Skeleton-agnostic animation compression with a learned entropy model

Nicholas Frechette, author of the Animation Compression Library, in an email reply to our questions about (1) ACL's design, (2) batch compression and decompression, and (3) neural codecs.

First and foremost, ACL is designed to be flexible and minimally invasive. Something easy to integrate into any C++ codebase. To that end I attempted to maintain a balance between decompression speed, memory footprint, latency, and compression speed. First, I think of what sort of codebase uses something like animation compression. By and large, animation decompression today happens on the CPU (and that was even more true 10 years ago when I began the project). Animation often is processed in two distinct steps:

• An animation graph/tree update pass (aka update pass) which advances time, reacts to gameplay changes (e.g. performs graph/tree topology changes through transitions), etc.
• A pose evaluation pass (aka evaluate pass) which samples clips, blends them, and performs any post-process type modifications (e.g. IK).

How data is accessed in both passes is quite different. In the update pass, we typically access two kinds of data:

• Data that is shared between multiple character instances (aka shared data). Data of this nature can comprise of things like graph/tree topology information, constant values a user has authored (e.g. play rate = 0.5), translation tables where an input is used to select among multiple outputs (e.g. for character type X, use sub-graph Foo), etc. This shared data is generally immutable at runtime and can be baked offline.
• Data that is unique per character instance (aka instance data). This would be actual node state, things like the current/previous evaluated type, previous condition values, etc. This instance data is generally read/write at runtime and is either instanced up front in bulk for the whole graph or on demand.

During the update pass, we are latency bound. Shared and instance data is, for the most part, randomly accessed and the processor will not be able to effectively hardware prefetch any meaningful amount. In this pass, it is thus critical to touch as little memory as you can to avoid evicting memory that was expensive to bring in (through a random access cache miss).

The evaluation pass is quite different, it is throughput bound because data is largely accessed in bulk and linearly. Things like decompression and pose blending touch a lot of memory but they do so with a predictable memory access pattern which means the processor can more easily hide latency and prefetch ahead. The nature of the code also means we can fill the CPU pipeline more easily as we perform the same work across many elements in loops. As a result of this, it is less important here to minimize how much memory is touched, but it can still matter.

Crucially, many animation runtimes use the same graph/tree topology structures for both passes (e.g. calling a different virtual function per node, one for each pass) while others use a different representation for the evaluate pass by building something akin to a GPU command buffer during the update pass (effectively linearizing the graph traversal). Which one your runtime uses matters because in the first instance, we will need randomly accessed instance (and perhaps shared) data while in the second form we might not (as the structure is linearized). Animation Blueprints in UE4 and UE5 use the same graph/tree topology while the new Unreal Animation Framework (UAF) is moving towards a command buffer approach.

ACL is designed to perform well in both forms of runtime. To that end, it aims to touch as little memory as possible to avoid evicting precious CPU cache data that might be needed (e.g. topology data). To achieve this, ACL is stateless and close in spirit to texture compression (e.g. BC7) where the compressed format is used as-is in memory without an intermediate unpacking step. It also means that nothing is cached and re-used frame to frame to again minimize the per clip instance memory footprint and avoid polluting the CPU cache. This ensures that once decompression is done, as little memory was evicted from the CPU cache as possible ensuring that the caller can still run at full speed, avoiding the need to reload the data it needed (that might have been evicted). This is especially important on mobile devices and Nintendo Switch where the CPU cache can be quite small and memory latency high.

That being said, we can evaluate the performance of animation compression along a few properties.

Memory footprint is always important because now more than ever, size is king when it comes to performance: the less memory you touch, the faster code tends to execute as memory is often the slowest thing happening. Memory is still precious on mobile/Switch as well even with modern games.

Compression speed is still important. Games today have more animations than ever before (recent games have over 100k clips for example). This means that importing new clips needs to be quick and predictable, cost wise. Bulk re-compression is uncommon, but it does happen. With good caching, this can be mitigated but care must still be taken. Long animations can sometimes take very long to compress. Old UE5 codecs could take up to 45-90 minutes to import/compress a 1-2 minute long cinematic (versus a few seconds with ACL).

Decompression speed is probably the most important part because animation generally runs on the CPU and animation is often one of the slowest parts of the frame on the CPU. It isn't uncommon for a character to use 3-20 clips to produce a final pose depending on how their locomotion is set up (e.g. motion matching versus blend spaces). This also makes decompression latency a critical aspect because gameplay often needs to wait for animation to finish (e.g. to update bounding boxes for spatial queries/ray casts and to update the physics simulation, and to run secondary motion). Games that run at 30 FPS will have more time to produce a frame but it means that it must react to the latest input more quickly as it happened a long time ago by the time the image gets on screen. Meanwhile a 60FPS or 120FPS game could take an extra frame to react to input and that might be fine (depending on the game, twitchy FPS games might need to react within the same frame still).

These points mostly apply to individual characters being animated, largely because they tend to be the most expensive to update. However, there are other things that can be animated that might not need the same speed/latency constraints (e.g. props in the environment, crowds). These might also lend themselves better to GPU evaluation.

ACL was thus meant to be a decent middleground. Something you can drop in and use quickly that would perform well enough to not be something you have to worry about too much afterwards. I think it has been quite successful in that regard (almost no one reaches out to me despite broadly being in use in the industry).

That should broadly answer your #1 point.

For #2, batch compression is a less relevant metric. Compressed animation data is generally cached in editor/offline and so while compression isn't unusual, it tends to occur for individual clips (e.g. on import, after a sync from p4). Batch decompression may be a useful metric, but for niche use cases. In my experience, it is very difficult on the CPU to batch animation decompression. Every single runtime I've seen produces a final pose per character one at a time and doesn't interleave the process (e.g. decompressing every first, then blending, etc). That might be more readily feasible for a fixed function animation runtime, something built for simple behavior (e.g. a crowd with a few nodes). These would be better suited to GPU evaluation as well. Bulk decompression would also require allocating up-front memory for the output pose and that can be expensive. It becomes a cpu/memory tradeoff: it is faster to bulk decompress, but it uses more memory.

For #3, I have seen neural networks that produce good animation fidelity, comparable to ACL, with decent decompression performance on the CPU, enough to be competitive (e.g. comparable to motion matching) and good memory footprint. This can be achieved with minimal latency when evaluated on the CPU. However, due to the model size, they tend to evict the entire CPU cache (sometimes both L1 and L2) which means they might slow down the caller (e.g. the evaluation pass). I haven't seen by how much yet. This might be okay if the animation runtime uses a command buffer approach (as I mentioned earlier). For this reason, I think the best performance measure you could get is of the entire evaluation pass, not just decompression, to capture the full impact on the CPU cache. You would have the same sort of behavior on the GPU with respect to the cache (I imagine).