n64js

Wow, yet again it's been ages since I posted. I just wanted to share some information about a project I've been tinkering with in my spare time - I ported Daedalus to JavaScript. You can try it out here: http://hulkholden.github.com/n64js/.

Be warned - even though it's based on Daedalus, a lot of features are unimplemented, and there are a whole host of new bugs that need fixing. It's also pretty slow (but not as slow as you might think, given that it's written in JavaScript).

I'm going to write up some notes about the implementation over at http://n64js.blogspot.co.uk/. Otherwise I'll probably see you back here in a few years ;)

Yes, I am alive

No, I didn't choke on a mince pie :)

I'm sorry for the lack of updates, both to the blog and to Daedalus. I've been working on Daedalus for about 10 years now, and those that have followed the project from the beginning will know that I have fairly regular periods of 'radio silence' followed by bursts of activity. I have lot of inertia, so once my interest shifts elsewhere it can take a long time to get me back on track. That's just the way I am.

This time around it's because I've been really busy with work. I don't want to go into specifics, but for the past couple of years I've been working on a high profile game, and since January most of my energy has gone into helping to get it finished.

I am unbelievably out of touch with the PSP scene. Recently, Kreationz got in touch to tell me about the progress that has been made with DaedalusX64. I was really pleased to hear that the project was being actively updated, particularly because I have a lot of respect for the people involved. The whole reason for making Daedalus open source was to give people the opportunity to learn from the source code and update it as needed, so it's great for this to be happening.

I can't guarantee that I'll have the time to actively help with DaedalusX64, or even that I'll be able to regularly update this blog. I thought it would be a good idea to give my public backing to DaedalusX64 so that the team can get on with rolling out updates. I probably won't be making a PSP release of Daedalus any time soon.

For what it's worth, I'm currently working on a new port of Daedalus. I'm going to keep the target platform secret for now, but I think it's very exciting. At the very least it means that I'm working on Daedalus again - I'm hoping to keep merging changes from DaedalusX64 back into the Daedalus source, and hopefully provide DaedalusX64 with one or two fixes of my own.

That's all for now. Apologies in advance if the next blog post is in December 2009 :)

-StrmnNrmn

Yuletide Update

It's been a while since the last update so I wanted to give some news on how things are going with work towards the next release.

I've spent a lot of time working on getting the HLE audio code working on the Media Engine. I've been making steady progress, but it's been taking longer than I initially expected. Fortunately, I'm very close to getting all of the audio processing moved over to the ME - in fact I believe I have just one significant bug left to fix.

The issue seems to be a very odd synchronisation bug which causes the emulator to lock up when running the audio processing asynchronously. As with many of these types of bugs, it's proving quite hard to track down because as soon as I change the code to debug the problem, the issue goes away. A true Heisenbug :(

What's particularly annoying is that the bug is stopping me from measuring how much of a difference running the audio code on the ME makes. Hopefully I'll be able to fix the bug over the Christmas break and be able to publish some timings over the new year.

As part of this work, I've also been writing a general-purpose 'job manager', which coordinates batches of work between the main CPU and the ME. The idea is to build on top of J.F.'s MediaEngine.prx to provide a simple interface for queing up and dispatching work asynchronously. When a job is added to the queue, a flag indicates whether the job is suitable for running on the ME, or whether it should just be run asynchronously on the main CPU instead.

Initially just the audio processing will run through the job manager on the ME, but eventually it should be possible run other pieces of work asynchronously too. I'm hoping that it will eventually be possible to move parts of the HLE graphics processing to run asynchronously too, but I need to investigate things a bit more first. That's a job for future releases however.

Anyway, that's all for now. I'm off to eat mince pies and watch The Great Escape on tv. Merry Christmas everyone :)

-StrmnNrmn

R14 Progress

It's been a while since I talked about R14 so I wanted to post a quick update on what I've been doing.

The Media Engine work has been going well. The job manager I talked about last week is now fairly functional and handles executing the audio upsampling code in 3 different modes: synchronously, asynchronously on the main processor, and asynchronously on the ME.

It's taken me a little longer to get the audio upsampling code working smoothly on the ME. I decided to focus on this initially (rather than Azimer's Audio HLE code) as it's a lot simpler and more self contained, but getting it working on the ME without any glitches required a little bit of work. I had to rewrite the simple ring buffer I was using to be lock-free. This is straightforward when dealing with a single reader thread and a single writer thread on the same processor, but a little more care is required when the reader and writer are operating on separate cores without cache coherency. I think getting this running glitch-free has helped prepare me well for the bigger task of getting Azimer's HLE code running asynchronously on the ME. I'll be working on this next.

Besides the ME work, I've had an interesting diversion getting some new font rendering working in Daedalus. I saw on the ps2dev.org forums that BenHur had released a library for rendering text using the PSP's built in fonts. I've always been a little unhappy with Daedalus's text rendering, and thought this would be a good opportunity to improve it. Here's a screenshot of the UI using BenHur's intraFont library (click through for a better-looking unscaled version):




I think this is looking a lot better than the previous font. The drop shadows really help make the text more readable. I also support multiple fonts for the first time, so the header text actually looks like header text :)

-StrmnNrmn

And the winner is...

This has been one of the hardest posts I've had to write, because I know whichever entry I pick as winner of the icon competition, a lot of people are going to be disappointed that I didn't pick their favourite.

Trying to select a winner has forced me to think about what I wanted to get out of this competition. On the surface, the main goal of the competition was just to replace the ugly default icon on the PSP's XMB. However as I've checked out all the entries over the past couple of weeks, I've come to realise that the main objective of the competition should be to define an identity for Daedalus.

In the end, one of the main criteria I've used to select the winner is how well the design works as a logo, or identity, for the project as a whole. For this reason I've picked Patrick Ahmann's design:




I think Patrick's design is wonderfully clean and stylish, and the colours look especially crisp and vibrant on the PSP's screen*. The typography, particularly for the 'daedalus' logo is very simple and distinctive and I think it would work well as a logo on the web. What's more, I got quite excited about using the cog motif on the logo as a progress indicator on loading screens :)

So, well done Patrick!

I want to say a big 'thank you' to everyone who contributed. Many people put a lot of work into this and I'm really sorry that I could only pick one winner. I think the quality of the final 10 was excellent, so if you disagree with my choice hopefully there's something there for you :)

-StrmnNrmn

*annoyingly Blogger insists on downsampling the images above slightly, so they look a little fuzzy. Click through for the original versions...

Icon Competition - Shortlist

I've just finished compiling a shortlist of what I think are the 10 best submissions for the icon competition.

I received just under 200 entries in total; I've been blown away by the response. Picking just 10 entries has been very difficult. In the end I had a 'longlist' of around 30 entries which I thought were superb, and it's taken me over an hour to whittle that down to just 10 entries.

Here is my shortlist, presented in the order they were submitted. Feel free to discuss these in the comments - I'll be reading your comments over the weekend, and I'll pick a final winner on Monday.

Thanks to everyone who entered - you're awesome!



-StrmnNrmn

Cladil

Pleaser

Pharonyk

Steven La

Victor Aguirre

Joël Dos Santos

Jay Mc

Patrick Ahmann

Hykypoo

juniorslick363

Edit 20/11/2007 12:00 Fix Pharonyk's name.

Last call for icons+backgrounds

Just a reminder that today is the last day for sending in your entries to the icon competition. I'll be making the shortlist sometime tomorrow morning, so provided you get your entry to 12pm (that's noon - GMT!) on Saturday it will probably be included.

Once I've made a shortlist of what I feel are the 10 best submissions, I'll post them up here so that people can give their thoughts, before I pick a final winner on Monday.

-StrmnNrmn

Media Engine progress

Over the weekend I described my plans for getting audio list processing working on the PSP's Media Engine. I'm making some decent progress so far. I've got Daedalus loading a kernel mode PRX to handle the ME nitty gritty, and I've managed to execute some test code on the ME successfully.

I've spent some time reviewing the audio code, trying to figure out if any bits are particularly amenable to running asynchronously, and trying to figure out if there is anything that is going to cause any problems when running this code on the ME. Fortunately it looks like all of Azimer's audio code is very straightforward C so there should be no problems getting it running on the ME once the synchronisation issues are dealt with. I've also realised that alongside the audio list processing there is also some expensive 44kHz upsampling code which will run very nicely on the ME too.

I have the feeling that debugging code on the ME is going to be particularly painful, so I want to try and catch as many of the obvious synchronisation bugs as early as possible. This evening I've started writing a job manager to 'simulate' executing code on the ME. The manager simply creates a thread which sits and waits for jobs to come in, mimicing the behaviour of the mediaengineprx. Once I've got the audio list processing running correctly through the job manager, I can easily switch things over to get these jobs running on the ME in parallel to the main core. That's the plan, anyway :)

-StrmnNrmn

Icons update (part 2)!

..And the last update today (I promise!)

I just wanted to say that I've now received about 100 entries for the icon competition. I'm about halfway through the list so far; apologies if I've not responded to your entry yet.

I've had a lot more entries than I was expecting. Originally I was planning on posting all the entries so that people could comment on their favourites before I picked a winner. There are too many entries for me to be able to do that, so next weekend I'll pick my favourite 10 and show them instead.

I've had lots of entries from readers of PSP-Generation, so a big 'merci beaucoup' for that. It's certainly helping exercise my rusty French :)

There's just under a week left before the competition ends, so be sure to get your submission in shortly. Thanks for all the entries so far!

-StrmnNrmn

PS Remember it's 'daedalus' and not 'deadalus'. It's an easy mistake to make :)

Media Engine

Earlier I discussed my plans for getting Daedalus's audio processing working on the PSP's Media Engine.

As I mentioned in that post, it's not just a case of changing some compiler setting to get this working. I've not spent much time investigating the ME so I may be wrong on a few of these points, but here are the current issues that I think need solving.

Firstly in order to access the ME I need to be running in kernel mode. This requires either running Daedalus in kernel mode, or (preferably) creating a kernel mode PRX that encapsulates the required functionality. I think kernel mode rules out anyone running with v1.50 firmware (hence my earlier post - please respond to the poll if you haven't already done so!) Maybe one of the more savvy psp developers out there can correct me on this? If no-one is using v1.50 any more then maybe it isn't even an issue.

Another problem is that although the ME is essentially the same processor as the main core, it has a different memory map. This means that things like the VRAM is invisible to the ME, so any code ported to run on the ME would have to be written to operate on main memory. This isn't an issue for Daedalus's audio list processing, but it would cause problems if I wanted to move display list processing to the ME too.

Touching on the memory map issue, another problem is the lack of cache coherency between the two cores. I need to be careful when accessing the same areas of memory with both cores to correctly flush and invalidate the data caches. Ideally any shared memory should be kept to a minimum, but this is easier said than done when porting existing code, rather than writing new code.

For a similar reason, any code which needs to run on the ME should avoid making any calls to the runtime library, including doing any system memory allocation. System calls are also ruled out. This is fairly easy to guarantee if you're writing new code, but again, it's a lot harder if you're porting existing code.

I think that's most of the issues from the hardware side. There are also a number of issues to be solved to do with the way that Daedalus handles audio and display list processing.

On the N64, the audio and display lists are processed asynchronously by the RSP coprocessor. In Daedalus, I can identify when these tasks are queued up for the RSP, intercept them, and process them synchronously (using high-level emulation rather than simulating the RSP execution directly).

The key thing here is that as far as the emulated N64 is concerned, audio and display list processing currently happens instantaneously. As soon as it kicks off the RSP it gets a interrupt to inform it that processing has completed. The whole process is very deterministic and I'm worried that by processing these display lists asynchronously on the ME that a number of intermittent and hard-to-debug issues will crop up. On the other hand, processing these tasks asynchronously is much closer to the behaviour of a real N64, which may fix some timing-related issues. It will also allow Daedalus to exploit the inherent parallelism that N64 roms were designed to take advantage of.

My current plan for ME audio support in R14 is:

1. Create a kernel mode PRX and get Daedalus successfully loading and invoking functions (under all supported firmwares). I've just about done this.
   
2. Add the code to support initialising and running code on the ME to the PRX. Test invoking user mode functions from the main EBOOT.PBP. I'll probably be using J.F.'s great sample code as a reference for this. Thanks J.F.!
   
3. Rewrite the audio list processing code so that it can be invoked synchronously or asynchronously as required (via some kind of configuration option). When running asynchronously it can just be run from a separate high-priority thread to start with. I can use this to test for various synchronisation issues without going through the pain of trying to do this on the ME first.
   
4. Audit the audio list processing code to minimise any memory accesses or ensure that they are correctly synchronised with the main core/thread. Any crt or system calls need to be eliminated or abstracted away (e.g. printfs NOP when compiled to run on the ME).
   
5. Invoke audio list processing code from the ME.
   
6. Cross fingers.
   

So, that's the plan; I'll keep you updated on my progress. If anyone has any experience doing this kind of thing on the ME it would be great to hear your thoughts.

-StrmnNrmn

R13 Issues, R14 Plans

Over the past week I've started making plans for what I want to do for R14.

To start with, R13 introduced a couple of issues which I want to fix. Firstly, a number of roms now no longer work with dynarec enabled, or show odd behaviour. For instance, Aerogauge now finishes the race as soon as the countdown completes. I've tracked this down to one of the dynarec optimisations I added in August, where I optimise fragments which jump back to themselves. This should be a 'safe' transformation, so it suggests there's a bug somewhere in my implementation. If I can't fix the bug in time for R14, I'll add a temporary setting to allow this optimisation to be disabled on a rom-by-rom basis (much like the 'dynarec stack optimisation' setting).

Secondly, it looks like something I changed for savestate support has broken the 'return to main menu' option. I added some logic to help ensure that when taking a snapshot for the savestate, the CPU is paused in a 'safe' state (i.e. no dynarec code is executing, nothing is running on the RSP, and nothing is executing in the branch delay slot.) It looks like I've messed something up which is causing the 'return to main menu' option to wait for a safe state before bailing out to the menu. Should be an easy one to fix.

Morgan suggested a nice idea in the comments, which is that I generate a thumbnail for the savestate as it is created to display alongside the slot in the UI. It's a little tricky to implement, as by the time the emulator is told to create a savestate, it has already obliterated the n64's framebuffer with the Daedalus UI. I'll have to do something quite clever like speculatively copy the n64's framebuffer into system memory every time you enter the Pause Menu, or create the screenshot on the first frame rendered after saving. Either way, I'd like to add this simple feature to R14.

Next on my list for R14 is to look at making more significant performance improvements. Over the months many people have been asking when I'd get around to implementing audio on the PSP's Media Engine. I've talked about this before, but always kept putting it off in order to work on easier optimisations.

The Media Engine is a bit of unknown territory for me. Even though it's practically identical to the main CPU, you can't just change a setting an suddenly have your code running on it. There are a number of small hurdles I have to overcome before I can get audio working on the ME, but this is my big goal for R14 (I'll save the technical discussion for the next post.) If all goes to plan this should mean that audio can always be enabled without a significant impact on framerate.

So in summary for R14: a few bug fixes, thumbnails for savestates, and audio without affecting framerate.

-StrmnNrmn

Firmware poll

I'm interested in which firmware everyone is running on. It would be very helpful if you could reply to this post with a) which model of PSP you're using ('fat'/slim, revision # if known) and b) which firmware you're using.

I have two PSPs. I have an original Japanese PSP with v1.00 firmware which I use for development, and a UK PSP which usually runs the latest official firmware (v3.72 at the moment.)

I'm interested because I want to know if anyone still requires the v1.50 (kxploit) versions of Daedalus that I release. I also need to figure out if it's worth my time getting a slim PSP for developing with the v3.xx+ firmware.

Thanks!
-StrmnNrmn

Icons update!

Well it's only been a couple of days, but I've received over 50 submissions to date - It's going to take me some time to go through them!

A couple of people have asked about the icon size. 134x74 or 140x80 are both fine.

Thanks for all the entries so far - there have been some really great ones. Keep them coming!

-StrmnNrmn

Icons for Daedalus

I've never taken the time to add icons to the Daedalus EBOOT.PBP. Mostly it's just because I've been lazy (when I develop Daedalus I usually run it through psplink rather than the XMB, so I rarely see the frontend), but also because I've never found a suitable set of icons to use.

There are plenty of icon/background packs floating around for Daedalus, but I've been reluctant to use any of them; I need something that I can freely distribute, but also something that doesn't infringe on any trademarks.

I'd like to add icons and a background to Daedalus for the next release, so I'm opening a 'competition' to try and find the best design. The 'prize' will be full credit in the release notes and on the Daedalus About screen (including a link to your website if you wish). Here's what I'm looking for:

• Background 480x272, preferably in .png format
  
• Icon 134*74, preferably in .png format
  
• No use of Nintendo's trademarks! This means no use of the "N64" or "Nintendo" logos. Sorry. Be inventive!
  
• You must own the work you submit, and give me permission to use it with Daedalus PSP
  

Email your submissions to me (strmnnrmn@gmail.com). I'll post all the submissions I receive by 16th November on this site to get people's thoughts, and make a decision the following Monday (19th November - that's two weeks today!)

Of course if you don't like whatever decision I make, you're always welcome to repack the Daedalus EBOOT.PBP and use any of the other graphics people send in :)

Time to get mspaint fired up!

-StrmnNrmn

Daedalus PSP R13 Released

I've just finished packing up R13 and uploading it to sourceforge:

Daedalus PSP R13 for v1.00 Firmware
Daedalus PSP R13 for v1.50+ Firmware
Daedalus PSP R13 Source

The most significant new feature is savestate support. You can now save your progress at any point, via the Pause Menu (accessed through hitting the Select button). Savestates are written out to the memory stick, and consume around a megabyte per slot. You can load up a savestate at any time from the Pause Menu, or via the front end (hit the right shoulder button to swap from the rom list to the savestate list.)

Other than savestates, the most significant change in R13 is a number of optimisations to the dynarec core which should give a 10-20% speedup depending on the title being played. I've tested these optimisations as much as I can, but if you find that roms which worked with R12 are now broken, try disabling 'dynamic recompilation' and/or 'dynarec stack optimisation' from the rom's preferences screen.

I haven't looked at compatibility at all in R13, so it's unlikely that any roms will have started working in R13.

I'm interested to hear your feedback on both of these features. Let me know if you have any problems with savestates, or if you've found roms are no longer working in R13. I'll try and keep on top of the comments pages over the next couple of weeks.

It's taken a LOT longer to get R13 out than I had hoped. I can't quite believe the last release was in June! I'm hoping now I've got this release out of the door I'll be able to get back to making small, more frequent releases.

-StrmnNrmn

Update A couple of things have cropped up in the comments. Firstly, when you create a savestate it can take a while - up to 10 seconds - during which time the screen is black. It will complete though - please be patient :). I'll have a look at adding a progress meter or something like that for R14.

There are a few titles that don't seem to be working well with some of the dynarec optimisations I added. If you're having problems, try turning off the 'dynarec stack optimisation' as discussed above.

Daedalus R13 Soon

I've just about finished implementing the last couple of savestate features I talked about last week.

I've got a bit of polishing to do and a little bit more testing, but I'm hoping I should be able to release R13 sometime tomorrow (Sunday).

I'll post again then with a complete list of changes and links to the latest builds for you to download. See you then!

-StrmnNrmn

R13 close, honestly!

Apologies again for the lack of updates. I know people get nervous when I don't update regularly, but I don't like to post when I have nothing new or exciting to talk about.

I'm very close to releasing R13, I've just been struggling to find the time to add the final finishing touches to the savestate support. I've found it hard to get into a regular working pattern since moving, so what should have been a 1 week job has ended up taking a month. Bioshock and Halo 3 haven't helped either.

Savestates are working very well. The current implementation provides 10 slots which are shared across all roms. You can save to any slot at any time from an option on the Pause menu, or choose to reload a previous savestate. In this way it works just like QuickSave/QuickLoad found in various PC games (I'm tempted to add a special slot for this to the top level of the Pause menu).

I use zlib to compress the savestates, so an 8MiB savestate compresses down to 1MiB (or even smaller if you're running games which don't use the Expansion Pak).

There are just a couple of things I need to finish off now before I can release. Firstly I need to check if the savestate you're loading is for a different rom than is currently loaded. If this is the case I need to scan through your available roms looking for the correct one, and load it. To make this efficient I need to get the RomDB which I use for the PC build working on the PSP.

The second thing I need to do is come up with a decent way of linking some metadata to the savestate so I can show this in the UI. Just simple stuff like the full name of the rom, time the savestate was created and the total time spent playing when it was created. I figure without this information it'll be quite difficult to figure out what it stored in each savestate slot. The alternative is to add a text entry system to the UI so you can name each savestate, but I think this will just delay things even more.

In summary R13 is really close and I've not forgotten about it. Just a few more things to sort out and it'll be ready for release.

-StrmnNrmn

Daedalus and the PSP Slim + Lite

A number of people have been asking if Daedalus will support the PSP Slim and Lite (PSP-2000), and if it does, if it will take advantage of the extra 32MiB RAM to improve the speed of emulated roms.

I am planning on supporting 3.xx firmware, but not until I have a PSP with suitable firmware which I can test with directly. People will probably end up compiling the Daedalus source for 3.xx (and they're welcome to), but until I can verify everything directly myself these will have to remain 'unofficial' builds. If there is strong demand for a 3.xx version of Daedalus then I'll consider setting up a donations page with the aim of buying a Slim and Lite for development.

As for the additional 32MiB RAM that the PSP-2000 provides, it's unlikely that this will provide for faster emulation. I've spent a lot of time reducing Daedalus's memory requirements so that it runs comfortably in the 32MiB that the PSP-1000 provides (it's actually a fair bit less than this when you take out OS overheads etc.)

There are only two main areas of the emulator that would benefit from an increase in cache size. The first is the texture cache, which is only used when video memory is exhausted. I fixed a few bugs with this months ago which means that very few roms ever have to place textures in system memory.

The second place that would benefit from a cache increase is the ROM cache. In order to support roms larger than the PSP's system memory, I dyamically map pages of the ROM into the PSP's RAM on demand (rather like virtual memory on PCs.) If a page of the ROM that is requested isn't in the cache, I have to pause emulation as I load it from memory stick. Currently this cache size is 2MiB. It could be increased up to 32MiB which would fit most N64 roms comfortably, and would entirely eliminate paging while the emulator is running. This might give a small speedup, but I don't believe this is currently a significant issue even with the tiny 2MiB cache that is currently in use.

Increasing the dynarec buffers would be very unlikely to provide a speedup, as I've yet to see a rom where they fill up before they are flushed for some other reason (typically instruction cache invalidation.)

So in summary, yes, I will support the PSP Slim and Lite at some point, but no, it's unlikely to offer much of an improvement over the PSP-1000s (other than being slimmer and lighter, obviously :)

-StrmnNrmn

Back Online

Apologies for the lack of updates. I've not had internet access at home since moving flat at the end of August. My ADSL was activated today so I'm finally up and running again. I can't believe how long it's taken!

Between the flat move and a busy few weeks at work (getting ready for the Tokyo Game Show) I've not had that much time to work on Daedalus, but I have made some progress on a really cool feature people have been asking for for some time, namely savestate support.

For those not familiar with savestates, the basic premise is very much like 'hibernate' or sleep modes on PCs and laptops. A savestate can be created at any point during emulation. At a later time the savestate can be reloaded to restore the emulator to the exact state it was in when the savestate was created.

What's really useful about savestates is that they work even if the underlying save mechanism used by a rom isn't supported properly. I'm hoping that by adding savestate support Daedalus will become significantly more usable for many roms. Savestates also make development a lot easier as it means I can jump straight into the middle of a game when debugging/profiling rather than sitting through the title sequence dozens of times.

I still have a little bit of work left to do on the savestate system, mostly on the UI side, but there are a number of additional optimisations I want to finish off before I release R13. I'm currently planning to get everything ready for release on either the weekend of the 29th Sept, or possibly the first weekend in October.

-StrmnNrmn

Whoops

It turns out that exophase had already suggested two of the three optimisations I implemented this weekend in a conversation we had way back in February.

Whoops! It's been such a long time, I'd totally forgotten we'd talked about it. I really should pay more attention sometimes - if I had been you might have seen some of these improvements way back in R9. Sorry exophase - kudos for spotting this so long ago!

-StrmnNrmn