Our fine-grained management is single-entry eviction from our code cache. Since profiling is expensive, the best way to do this is to use a simple FIFO circular buffer, which has been shown to have miss rates comparable to more complex schemes like LRU or even LFU.

This replacement strategy means that repeated evictions will replace the entire cache, repeatedly, so we need to make sure the cache is large enough. But there's no reason to make it so large that it holds every instruction the application ever executes: we only need to capture the working set of the application. The working set varies by application: for some it's a small fraction of total code size, for others nearly every instruction executed is hot. This is the heart of our algorithm: identifying when the cache needs to be resized.