I'm in the proces of rewriting the dynamic recompilation engine for Daedalus. The previous version just about did the job on the PC, but the approach I originally used had a number of problems when it came to the PSP.
Firstly it was a huge memory hog. As memory is relatively cheap on the PC, the recompilation engine is fairly enthusiastic about recompiling blocks of code almost as soon as it comes across them. I'm not so sure this is such a good strategy now, as thousands of ops are recompiled never to be executed again. On the PSP I just can't afford this memory overhead.
Another big problem with the PC dynarec is that there's currently no way for me to easily flush old fragments of code from the cache to save memory. Again this isn't much of a problem on the PC, but on the PSP I can't afford to have old fragments of code lying around, chewing up valuable memory.
Finally the PC version of the dynarec is sufficiently old, poorly documented and overly complex that it's almost impossible for me to improve and properly debug. In general I'd rather refactor systems than rewrite them from scratch, but I feel there are so many drawbacks with maintaining the old system that it makes sense to look for an alternative solution.
Earlier in the week I started doing a bit of background reading on dynamic recompilation. In the past I've always struggled to find any useful papers on the subject (pehaps because 'dynamic recompilation' seems to be emulator-scene terminology). Googling 'dynamic code generation' turned up this paper:
Dynamo: A Transparent Dynamic Optimization System
Although this is talking about native recompilation (i.e. interpreting and recompiling a binary on the same platform it was developed for), it addresses the issues I raised above. What I find incredibly impressive is that they can improve on the speed of native execution, even with the overhead that they're generating by initially interpreting the code. I feel that if I can use their approach for a new version of Daedalus's dynamic recompiler I'll be able to obtain significant improvements in speed and memory usage over the old version.