It's been a couple of weeks since the last update. As people tend to get a bit nervous when I don't update regularly, I just wanted to let everyone know that I'm ok and still thinking about getting R12 ready :)
I'm still in the middle of looking at fixing the various dynarec bugs which are keeping Super Smash Bros from working with Dynarec enabled. I've not made as much progress as I would have liked since the last post, as I've been very busy with work and various other commitments. Hopefully I should get a bit more time to work on R12 from the end of this week. I'll post an update as soon as I have more news to report on.
-StrmnNrmn
Tuesday, May 29, 2007
Tuesday, May 15, 2007
Dynarec Fixes
In the previous update I mentioned that the dynarec wasn't working with Goldeneye:
I've spent a couple of evenings investigating this problem, and I've finally had a bit of luck in determining the root cause. As I suspected it was due to a bug in the PSP dynarec implementation - it turns out that I was fiddling addresses to compensate for the different endianness of the PSP twice (essentially having the same effect as not compensating in the first place).
The fix was very easy to implement (just a single line of code). It's quite a significant bug, so I'm hoping that it will fix dynarec issues with a few other roms too. Unfortunately it looks like the bug which is preventing dynarec working with Super Smash Bros. is a different issue, so I'll have to spend some more time investigating that.
I'm pleased to say that Goldeneye is now running fine with dynarec, so it now boots and runs the intro sequence very quickly. Although the intro and menu run at around 30-60fps, in-game is still very slow (around 5-8fps) so I need to profile this and see what's going on.
As I've hinted above, the next job is to have a look why dynarec is causing Super Smash Bros to hang. A cursory look revealed that it's still broken even when I disable most of the native code generation and directly call the interpretive instruction handlers from the compiled fragment. Because of this I think the problem has something to do with how traces are selected and the resulting fragments are linked together. I'll know more later in the week.
-StrmnNrmn
...I can still only get the rom running with dynarec disabled on the PSP, so the framerate is currently just 6-7 fps. Interestingly it runs fine with dynarec on the PC with the same set of changes, so this indicates a bug somewhere in the PSP dynarec implementation.
I've spent a couple of evenings investigating this problem, and I've finally had a bit of luck in determining the root cause. As I suspected it was due to a bug in the PSP dynarec implementation - it turns out that I was fiddling addresses to compensate for the different endianness of the PSP twice (essentially having the same effect as not compensating in the first place).
The fix was very easy to implement (just a single line of code). It's quite a significant bug, so I'm hoping that it will fix dynarec issues with a few other roms too. Unfortunately it looks like the bug which is preventing dynarec working with Super Smash Bros. is a different issue, so I'll have to spend some more time investigating that.
I'm pleased to say that Goldeneye is now running fine with dynarec, so it now boots and runs the intro sequence very quickly. Although the intro and menu run at around 30-60fps, in-game is still very slow (around 5-8fps) so I need to profile this and see what's going on.
As I've hinted above, the next job is to have a look why dynarec is causing Super Smash Bros to hang. A cursory look revealed that it's still broken even when I disable most of the native code generation and directly call the interpretive instruction handlers from the compiled fragment. Because of this I think the problem has something to do with how traces are selected and the resulting fragments are linked together. I'll know more later in the week.
-StrmnNrmn
Thursday, May 10, 2007
Goldeneye Update!
Ok, I know the last thing I said yesterday was:
I couldn't quite resist having a quick hack to see if I could get Goldeneye working using the third method I outlined. It turned out to be relatively painless to crowbar in the required changes, and I'm pleased to say that I've managed to get Goldeneye booting again! :)
With the changes, Daedalus now dynamically pages the Goldeneye ROM into the PSP's RAM so that it no longer requires a free 12 MiB. I now check the instruction pointer on every instruction fetch in the interpretive core*, and raise a TLB Read Miss exception if required. If this happens, it causes Goldeneye's exception handler to realise that the required code isn't in memory, and to load it in from ROM before continuing execution.
So here are the first screenshots of Goldeneye running under Daedalus R12 - you can see there are a handful of graphical glitches to sort out, particularly on the main menu:
Not bad for an hour's work!
There's still quite a long way to go though. Firstly, I can still only get the rom running with dynarec disabled on the PSP, so the framerate is currently just 6-7 fps. Interestingly it runs fine with dynarec on the PC with the same set of changes, so this indicates a bug somewhere in the PSP dynarec implementation. Secondly, I've not been able to get in-game yet - it just seems to hang with a black screen shortly after starting to load the level. I've not figured out whether this is just running very slowly without dynarec and taking its time, or if there's another bug to be fixed somewhere.
Getting the rom booting is a great first step though - it can only continue to improve from here :)
-StrmnNrmn
*I probably want to tidy things up to avoid doing this for most other roms as it can be quite expensive - especially if dynarec is disabled.
Anyway, given the work involved Goldeneye isn't an immediate priority, but I am thinking about it.
I couldn't quite resist having a quick hack to see if I could get Goldeneye working using the third method I outlined. It turned out to be relatively painless to crowbar in the required changes, and I'm pleased to say that I've managed to get Goldeneye booting again! :)
With the changes, Daedalus now dynamically pages the Goldeneye ROM into the PSP's RAM so that it no longer requires a free 12 MiB. I now check the instruction pointer on every instruction fetch in the interpretive core*, and raise a TLB Read Miss exception if required. If this happens, it causes Goldeneye's exception handler to realise that the required code isn't in memory, and to load it in from ROM before continuing execution.
So here are the first screenshots of Goldeneye running under Daedalus R12 - you can see there are a handful of graphical glitches to sort out, particularly on the main menu:
Not bad for an hour's work!
There's still quite a long way to go though. Firstly, I can still only get the rom running with dynarec disabled on the PSP, so the framerate is currently just 6-7 fps. Interestingly it runs fine with dynarec on the PC with the same set of changes, so this indicates a bug somewhere in the PSP dynarec implementation. Secondly, I've not been able to get in-game yet - it just seems to hang with a black screen shortly after starting to load the level. I've not figured out whether this is just running very slowly without dynarec and taking its time, or if there's another bug to be fixed somewhere.
Getting the rom booting is a great first step though - it can only continue to improve from here :)
-StrmnNrmn
*I probably want to tidy things up to avoid doing this for most other roms as it can be quite expensive - especially if dynarec is disabled.
Goldeneye
While I'm at it with the blog updates, I also invesitgated why Goldeneye no longer works in Daedalus (it did run with some of the very early releases, but stopped working some time ago.)
Goldeneye is quite an unusual game in that it's one of the few titles that executes code from virtual memory. I'm not sure exactly why it does this, but I suspect that it's to allow it to free up as much RAM as possible. It sets up a virtual memory range for the code, and pages in chunks of data to physical RAM as required (i.e. on TLB miss exceptions). I guess this means that it can just keep a few dozen KiB of code in RAM at a time, rather than the entire code segment.
Anway, emulating this accurately is incredibly slow - for every instruction that is executed, the emulator needs to check whether the instruction fetch would cause a TLB miss exception, and if so invoke the N64's exception handler. This is quite a time consuming process, so I tend to cheat and assume that the instruction pointer is in physical memory.
To get around this problem with Goldenye, I set up a couple of bogus entries in Daedalus' memory map that point directly to the correct region in the ROM image. In a way it's as if the N64 has a much larger TLB, and all the pages are permenantly mapped.
Normally (i.e. on the PC version of Daedalus) this all works fine. The problem as far as the PSP version is concerned is that the rom (all 12 MiB for Goldeneye) must be permenantly loaded into memory, and there just isn't enough memory left to do this anymore. As most N64 roms are simply far too large to fit in the PSP's RAM, I have to page chunks in from the memory stick on demand (pretty similar to what Goldeneye was doing on the N64 really :D)
So, there are a few possibilities for getting Goldeneye to work:
Of all of these, the first solution is probably the easiest, but it's a bit hacky (I'd have to allocate a 12 MiB for Goldeneye at startup, and carve this up for different subsystems if other roms were run).
The second solution is a bit less horrible, but I'm still not sure it's possible without checking the instruction pointer for every instruction that's executed (and paging in chunks of ROM as required).
That leaves the third solution which is probably the most work, but I think will be the most stable solution in the long run. It may also help a few other roms that are using similar techniques to Goldeneye to run correctly. If I can get it to work in conjunction with the current dynarec implementation, it shouldn't even be any slower than the current hack that's in place.
Anyway, given the work involved Goldeneye isn't an immediate priority, but I am thinking about it.
-StrmnNrmn
Goldeneye is quite an unusual game in that it's one of the few titles that executes code from virtual memory. I'm not sure exactly why it does this, but I suspect that it's to allow it to free up as much RAM as possible. It sets up a virtual memory range for the code, and pages in chunks of data to physical RAM as required (i.e. on TLB miss exceptions). I guess this means that it can just keep a few dozen KiB of code in RAM at a time, rather than the entire code segment.
Anway, emulating this accurately is incredibly slow - for every instruction that is executed, the emulator needs to check whether the instruction fetch would cause a TLB miss exception, and if so invoke the N64's exception handler. This is quite a time consuming process, so I tend to cheat and assume that the instruction pointer is in physical memory.
To get around this problem with Goldenye, I set up a couple of bogus entries in Daedalus' memory map that point directly to the correct region in the ROM image. In a way it's as if the N64 has a much larger TLB, and all the pages are permenantly mapped.
Normally (i.e. on the PC version of Daedalus) this all works fine. The problem as far as the PSP version is concerned is that the rom (all 12 MiB for Goldeneye) must be permenantly loaded into memory, and there just isn't enough memory left to do this anymore. As most N64 roms are simply far too large to fit in the PSP's RAM, I have to page chunks in from the memory stick on demand (pretty similar to what Goldeneye was doing on the N64 really :D)
So, there are a few possibilities for getting Goldeneye to work:
- Permenantly disable dynarec for Goldeneye, and reuse the dynarec buffers (6MiB) the expansion pak (unneeded anyway, 4MiB) and anything else I can scavenge, to fit the rom in a contiguous block in memory.
- Investigate a way of getting the memory-mapping hack to work with rom caching.
- Re-examine how I handle TLB miss exceptions for instruction fetches, and implement them correctly.
Of all of these, the first solution is probably the easiest, but it's a bit hacky (I'd have to allocate a 12 MiB for Goldeneye at startup, and carve this up for different subsystems if other roms were run).
The second solution is a bit less horrible, but I'm still not sure it's possible without checking the instruction pointer for every instruction that's executed (and paging in chunks of ROM as required).
That leaves the third solution which is probably the most work, but I think will be the most stable solution in the long run. It may also help a few other roms that are using similar techniques to Goldeneye to run correctly. If I can get it to work in conjunction with the current dynarec implementation, it shouldn't even be any slower than the current hack that's in place.
Anyway, given the work involved Goldeneye isn't an immediate priority, but I am thinking about it.
-StrmnNrmn
R11 Mario HUD Issue
As a few people have pointed out on the R11 release comments page, I broke the HUD in Mario 64 in R11 :(
I investigated a little, and it turns out it was due to a tiny change I made sometime on Saturday. Basically I changed the alpha threshold test from >= to >, and so nothing with an alpha of 0 is ever rendered. This shouldn't be an issue, except for the bizarre render states that Mario sets up to render its HUD.
I don't think this is a too significant issue as Mario is just as playable in R10. I'd rather release R12 a little early to fix this than mess around releasing an R11b version or whatnot. We'll see how things go with the SSB work.
-StrmnNrmn
I investigated a little, and it turns out it was due to a tiny change I made sometime on Saturday. Basically I changed the alpha threshold test from >= to >, and so nothing with an alpha of 0 is ever rendered. This shouldn't be an issue, except for the bizarre render states that Mario sets up to render its HUD.
I don't think this is a too significant issue as Mario is just as playable in R10. I'd rather release R12 a little early to fix this than mess around releasing an R11b version or whatnot. We'll see how things go with the SSB work.
-StrmnNrmn
Wednesday, May 09, 2007
Early SSB Progress
As I mentioned on Sunday, my main focus for R12 is to fix whatever's causing the dynarec to hang with SSB, but to start with I've spent a little time looking into fixing some of the graphical issues plaguing Super Smash Bros.
In the end I fixed four or five separate problems which were all causing quite significant glitches.
The first bug was that I was assuming that the fixed-point texture coordinates specified in the TexRect (i.e. 2d textured rectangle) commands were unsigned. It turns out that they're actually signed - SSB just happens to be one of the few roms that ever specifies negative values.
The next issue was that I was ignoring the RDP tile* mask parameters, and assuming that the tile width and height always defined the extents for both wrapping and clamping. As it turns out, the mask values (if set) define how many bits from the texture coordinate to accept - all the other bits from the texture coordinate are ignored. This means that if a tile defines a mask of 5, the texture will wrap every 2^5, or 32 texels. What's quite interesting about the N64 is that it lets you define different limit for clamping than for wrapping. For instance, you can wrap every 32 texels, but clamp the texture to a coordinate of 100 (i.e. after the texture has repeated 3 and a bit times.) It turns out this is exactly what SSB was doing.
A related issue was that as the PSP only supports textures with power-of-2 dimensions, I have to pad out non-power-of-2 textures from the n64 with empty space. As I was assuming that the n64 always clamped at power of 2 boundaries, it meant that the PSP occasionally displayed texels from bits of the texture that I hadn't initialised. To get around this I just manually repeat the last column/row of the texture when converting it on the PSP.
Another issue I fixed was relating to how I handled 'swizzled'** textures that were not an even number of texels. In the second screenshot on the left below, you can see a fringe of corrupt pixels down the right-hand side of each character's icon. This was due to me not correctly handling all of the texels the swizzled rows, but it only happened when the rows were not a multiple of 8 bytes (in the example below, the textures were 30x32 texels at 16bpp). Again, this is something SSB does that I've not seen in other roms.
I fixed a bug relating to how I was recolouring textures. In order to support some of the N64's more complex combiner modes on the PSP, I need to replace the RGB channels of a texture while maintaining the existing alpha channel. As it turns out, I wasn't handling this correctly for textures that weren't a power of 2. This is what is messing up the black background behind the 'Super Smash Bros' text on the first screenshot. Whoops.
Wow! That's quite a few bugs! Here's a few screenshots, with SSB running in R11 on the left, and the R12 work-in-progress on the right:
What's quite nice is that even though I've been fixing these with SSB in mind, there are likely to be lots of other roms which I've not tested that are relying on the same behaviour.
Graphically SSB is looking pretty slick, so now it's on to chasing down that dynarec bug :)
-StrmnNrmn
* The RDP has 8 tile attribute descriptors, which are used to tell the RDP how to interpret the data loaded into it's 4KiB of texture memory. They define properties such as the tile's format, width, height and whether to clamp or wrap and so on.
** On the N64, all odd rows in a texture have pairs of texels alternated (or more precisely pairs of 4-bytes of data.) The reason for this is that the N64's texture memory is composed of two separate banks of 2KiB, and organising texels like this allows for multiple texels to be accessed simultaneously when sampling from the texture. I use the term 'swizzled' loosely as it's quite a different process than swizzled textures on modern consoles (such as the PSP and the Xbox (see page 28)), even if it is performed for similar reasons.
In the end I fixed four or five separate problems which were all causing quite significant glitches.
The first bug was that I was assuming that the fixed-point texture coordinates specified in the TexRect (i.e. 2d textured rectangle) commands were unsigned. It turns out that they're actually signed - SSB just happens to be one of the few roms that ever specifies negative values.
The next issue was that I was ignoring the RDP tile* mask parameters, and assuming that the tile width and height always defined the extents for both wrapping and clamping. As it turns out, the mask values (if set) define how many bits from the texture coordinate to accept - all the other bits from the texture coordinate are ignored. This means that if a tile defines a mask of 5, the texture will wrap every 2^5, or 32 texels. What's quite interesting about the N64 is that it lets you define different limit for clamping than for wrapping. For instance, you can wrap every 32 texels, but clamp the texture to a coordinate of 100 (i.e. after the texture has repeated 3 and a bit times.) It turns out this is exactly what SSB was doing.
A related issue was that as the PSP only supports textures with power-of-2 dimensions, I have to pad out non-power-of-2 textures from the n64 with empty space. As I was assuming that the n64 always clamped at power of 2 boundaries, it meant that the PSP occasionally displayed texels from bits of the texture that I hadn't initialised. To get around this I just manually repeat the last column/row of the texture when converting it on the PSP.
Another issue I fixed was relating to how I handled 'swizzled'** textures that were not an even number of texels. In the second screenshot on the left below, you can see a fringe of corrupt pixels down the right-hand side of each character's icon. This was due to me not correctly handling all of the texels the swizzled rows, but it only happened when the rows were not a multiple of 8 bytes (in the example below, the textures were 30x32 texels at 16bpp). Again, this is something SSB does that I've not seen in other roms.
I fixed a bug relating to how I was recolouring textures. In order to support some of the N64's more complex combiner modes on the PSP, I need to replace the RGB channels of a texture while maintaining the existing alpha channel. As it turns out, I wasn't handling this correctly for textures that weren't a power of 2. This is what is messing up the black background behind the 'Super Smash Bros' text on the first screenshot. Whoops.
Wow! That's quite a few bugs! Here's a few screenshots, with SSB running in R11 on the left, and the R12 work-in-progress on the right:
What's quite nice is that even though I've been fixing these with SSB in mind, there are likely to be lots of other roms which I've not tested that are relying on the same behaviour.
Graphically SSB is looking pretty slick, so now it's on to chasing down that dynarec bug :)
-StrmnNrmn
* The RDP has 8 tile attribute descriptors, which are used to tell the RDP how to interpret the data loaded into it's 4KiB of texture memory. They define properties such as the tile's format, width, height and whether to clamp or wrap and so on.
** On the N64, all odd rows in a texture have pairs of texels alternated (or more precisely pairs of 4-bytes of data.) The reason for this is that the N64's texture memory is composed of two separate banks of 2KiB, and organising texels like this allows for multiple texels to be accessed simultaneously when sampling from the texture. I use the term 'swizzled' loosely as it's quite a different process than swizzled textures on modern consoles (such as the PSP and the Xbox (see page 28)), even if it is performed for similar reasons.
Sunday, May 06, 2007
Daedalus R11
Daedalus R11 is finally here. It took a little longer than I expected to merge the changes across to the sourceforge subversion repository but I got there in the end :)
Anyway, here are the latest files:
Daedalus PSP R11 for v1.00 Firmware
Daedalus PSP R11 for v1.50+ Firmware
Daedalus PSP R11 Source
Here's the changelist:
As always, let me know your thoughts on the comments pages. I don't always have time to reply to all of the comments, but I do read every one of them.
-StrmnNrmn
Anyway, here are the latest files:
Daedalus PSP R11 for v1.00 Firmware
Daedalus PSP R11 for v1.50+ Firmware
Daedalus PSP R11 Source
Here's the changelist:
- [*] Large overhaul of rom settings and preferences. All settings should now be persistant.
- [+] Native support for palettised textures for faster rendering and less memory usage.
- [+] Provided a fixed size pool for all texture memory, to limit memory usage.
- [+] Expansion Pak enabled by default.
- [!] Updated rom info database to include save mechanism for many roms.
- [!] Fix memory leak in texture cache.
- [!] Fixed conversion of RGBA/32bpp textures with odd alignments.
- [!] Fix crash which occurred when recolouring textures in low memory situations.
- [^] Large rewrite of texture cache to reduce memory usage and improve performance.
- [^] Various optimisations to the way textures are looked up in the cache and installed.
- [^] Optimise offsetting/scaling of texture coordinates.
- [^] Improve performance of dynarec fragment cache and reduce memory fragmentation.
As always, let me know your thoughts on the comments pages. I don't always have time to reply to all of the comments, but I do read every one of them.
-StrmnNrmn
R11 Sometime on Sunday
I'm very close to having R11 ready for release, and I'm aiming to have everything finished some time tomorrow morning (that's Sunday morning, UK time.)
I've spent the day putting in number of changes to the way that Daedalus allocates memory for various lookup tables. Namely, I've replaced a couple of stl maps I was using to keep track of cached textures and dynarec fragments with sorted vectors, which should help reduce fragmentation and improve cache usage. Basically it should make things even more stable and give another small speedup.
I've also been through and updated the rom.ini file I talked about a couple of weeks ago to make sure that it contains accurate information for the save type. I noticed that many of the entries for European roms were undefined, which can cause various compatibility problems. Hopefully this should mean that a few more European roms are working in R11.
Finally I realised that there was a dynarec issue which was preventing Super Smash Bros from going in-game. Although that's not fixed (that's something for R12) I have disabled dynarec for SSB by default. With my various texture cache fixes SSB is now running in-game, albeit with messed up graphics and at just 15 fps. Here's a teaser screenshot of SSB running in R11:
Given how many people have been asking for SSB compatibility to be improved, I'm pretty sure this is going to be my main focus for R12. If I can fix the dynarec issue it looks like this is going to run at a solid 25-30fps or more.
Anyway, check back sometime over the next 12 hours for R11 (I'm off to bed for now, let's hope I don't lie in :)
-StrmnNrmn
I've spent the day putting in number of changes to the way that Daedalus allocates memory for various lookup tables. Namely, I've replaced a couple of stl maps I was using to keep track of cached textures and dynarec fragments with sorted vectors, which should help reduce fragmentation and improve cache usage. Basically it should make things even more stable and give another small speedup.
I've also been through and updated the rom.ini file I talked about a couple of weeks ago to make sure that it contains accurate information for the save type. I noticed that many of the entries for European roms were undefined, which can cause various compatibility problems. Hopefully this should mean that a few more European roms are working in R11.
Finally I realised that there was a dynarec issue which was preventing Super Smash Bros from going in-game. Although that's not fixed (that's something for R12) I have disabled dynarec for SSB by default. With my various texture cache fixes SSB is now running in-game, albeit with messed up graphics and at just 15 fps. Here's a teaser screenshot of SSB running in R11:
Given how many people have been asking for SSB compatibility to be improved, I'm pretty sure this is going to be my main focus for R12. If I can fix the dynarec issue it looks like this is going to run at a solid 25-30fps or more.
Anyway, check back sometime over the next 12 hours for R11 (I'm off to bed for now, let's hope I don't lie in :)
-StrmnNrmn
Subscribe to:
Posts (Atom)