Translating Saturn Grandia

[zyrobs]

Have you checked the file bounds, as in, what happens when the script file is larger than the old one, and what happens if one particular text is long enough to push another one down? Are the script start points hardcoded pointers or some other control values that indicate "text #001 starts here"?

Those were the issues I came across in SOTN, text locations were fixed and all string values used hardcoded pointers (and the values pointing to them were also pointers). The whole script file was a compiled mess, somewhere in the middle of the game code so you couldn't even recompile and pad it out since it would get overwritten by other code, it had a fixed length.

[TrekkiesUnite118]

Have you checked the file bounds, as in, what happens when the script file is larger than the old one, and what happens if one particular text is long enough to push another one down? Are the script start points hardcoded pointers or some other control values that indicate "text #001 starts here"?

Those were the issues I came across in SOTN, text locations were fixed and all string values used hardcoded pointers (and the values pointing to them were also pointers). The whole script file was a compiled mess, somewhere in the middle of the game code so you couldn't even recompile and pad it out since it would get overwritten by other code, it had a fixed length.

I haven't yet. Though if I were to take a guess I'd bet that the game will handle the expanded text fine as I'm not seeing anything in the text that looks like pointers, but something else will go wrong later on. The files that contain the script seem to contain more than just the script. Again taking a guess it seems that these files are for each specific area of the game, and contain the script for that area along with other data. So I'd imagine that if we expand text that bumps everything else in the file down, there is going to be a spectacular crash somewhere else down the line.

I think this is where comparing the JP and EN PS1 files to figure out how they dealt with this issue will shed some light on what to do. Also comparing the JP Saturn MDT files to the JP PS1 MDP files would probably be beneficial as well as it can shed light on what is different between the formats.

EDIT:

So I think I might be figuring out how the script pointers work based on comparing the PS1 versions, and then spot checking with the Saturn version.

So here is what I believe to be one complete text routine from the file BA38.MDT at x05B4:



This highlighted portion I believe is a header of some kind. It seems to change from text sample to text sample, so there's probably optional values that are in some text boxes but not in others.


Now this next highlighted portion is the important part. This value I believe represents how many bytes until the end of this text box. In this example it's 5A:


If we then move forward 5A that will bring us to the end of this block of data. The end of a block of text data seems to be tagged with this possible footer:


After this the next text box starts. I've been able to keep following this pattern and consistently keep finding what appears to be the correct number of bytes and the start of the next text box.

I'm going to keep testing this but so far it's looking promising.

Also, another interesting thing of note is that it seems that the English PS1 versions files seem to be close to if not the same size as the Japanese PS1 files. It looks like Sony removed unnecessary padding to keep the files close to the same size from what I can tell.

[TrekkiesUnite118]

Ok, so I think I've cracked the code of the MDT/MDP file headers. This information seems to hold true for both the Saturn and PS1 files, the only difference being that in the PS1 files pointers are Little Endian where as in the Saturn files they're Big Endian.

• The header is 512 Bytes in size.
• The header seems to consist of just a pointer table
• Each entry is 8 bytes total in size
• The first 4 bytes seem to be the place in the file where the data lives
  
• The second 4 bytes seem to be the size of the data.
• The entry that tells us where the text lives and how big it is seems to be at x60 every time.
• There's an entry at x58 that seems to point to a 24 byte piece of data right before the text. Perhaps this is a header or something?
• Consistently I'm seeing in the English PS1 version that any addresses that come after the address defined at x60 have their values increased by the increase in size of the text data, and any values that come before it are untouched.
  
• I'm also seeing again padding being removed in the PS1 files of the exact amount added to the text.
  
• Another oddity I'm seeing is that the last entry in the table at x1F8 seems to also have it's size increased by the increase in text size. But I'm not sure what that value represents or what data was added.

Some other oddities I'm seeing is that some entries don't seem to make sense as the addresses are beyond the size of the file, but they're still updated in the English PS1 version.

With this info I imagine I should be able to make a program that can do the following:

• Parse the MDT files
• Extract the script portions
• Put the MDT file back together with a new script portion.
• Recalculate the pointer table.

The hard part to automate in code I'd imagine would be to find a portion of padding in the file that can be removed to keep the file size the same.

[TrekkiesUnite118]

So I wrote a program that allows me to parse the MDT and MDP files from the Saturn and Playstation versions. This basically strips the first 512 bytes and reads them as a pointer table. This information is then used to parse the rest of the file. Knowing that 0x60 marks the table entry for the script, I wrote some code to copy the bytes defined in the table and write them to a file. This has essentially allowed me to dump the raw script for the game out of all the MDT/MDP files.

I've made it so it also writes the pointer table to a file, as well as the pre-script and post-script sections to files. This way it should be easy to make some code to do the reverse and put all the pieces back together and recalculate the pointer table.

On to the more interesting stuff though. After dumping the script for the Japanese Saturn, Japanese PS1, and English PS1 versions, I discovered that the Saturn and Japanese PS1 script data is identical. The initial cutscene I was looking at appears to be a special snowflake where they've added a bunch of additional control codes in the PS1 version. After that, things start lining up between the two. The only difference is the non-script parts of the data are byte swapped due to endian differences.

This is making it really easy to start identifying pointer values and control codes by comparing the different versions. I'm still figuring things out in these files, but I'm starting to think it might be possible to write a program that could automatically convert the English PS1 script files into a format the Saturn is expecting.


I've attempted to attach some samples for people to look at if they wish. The extension says .SCRIPT but that is just something I came up with to keep things organized. They're just raw data that you can look at in a hex editor.

[TrekkiesUnite118]

So I think I might have some logic for how to do an automated conversion. Since the script format is the same, but with portions byte swapped, converting is straight forward from a manual process, but it's very time consuming. So automating it would probably save time, but it's a bit tricky. From what I've found of the script files, there's no real common pattern I can use confidently to know exactly where the the text portion starts, and where the pre-text header/control code portion is. Basically some control codes are optional and aren't always used which makes knowing what to scan for difficult.

But I do know the following 2 pieces of information seem to be consistent enough:

1) Each text sequence ends in the byte 0x07
2) There will always be at least a 2 byte sequence that will tell us the amount of bytes to the end of the text sequence, starting immediately after those bytes. From this point on is the portion that doesn't need to be byte swapped. Now this isn't always in the same pattern sadly. I've seen at least the following variations

0x2X XX
0x3X XX
0x9X XX

The X XX represents the size value in Hex. Sometimes with the 2 variant there is another 2 bytes of 0x3X XX that represents the size to the end of the sequence starting at the offset before those two bytes.

Now since we know the file format is identical between versions for the script data, and the control codes are the same, the only differences is the portions that precede and include the size bytes are byte swapped, and obviously the English text isn't the same size. Control codes appear to be the same. However for the Saturn version the control code 03 needs to be added at the start of each line to tell it to use the 8x16 font.

So with all of this in mind this was my thought process for a how it could be converted automatically:

1) Split Script file for both English PS1 and Japanese Saturn or PS1 files on byte value 07 to break it down into it's individual script sequences.

For English PS1 Script Pieces
2) Scan the script piece two bytes at a time, checking to see if the value of the low half byte of the first byte combined with the value of the second byte is equal to the remaining size of the piece when byte flipped.

*When doing the check, check the size starting at the offset immediately after these two bytes.

3) When we have a match, check the previous 2 bytes to see if they are also equal in distance from the offset immediately before them. If yes, make note of the value and it's offset.

4) Byte flip the piece from offset 0 to the last offset of the size byte.

5) From here on compare this piece to it's Japanese equivalent. Store size difference as a variable.

6) Scan byte by byte until we don't match, the data prior to this mismatch should be control codes. At this point insert 03. Keep a count of each 03.

7) If Japanese file is larger, jump English offset ahead by Size difference. If English is larger, jump Japanese offset ahead by size.

8) Scan until we do get a match. We're Hoping that we've found a control code, so check that the matched value starts with 0.

9) If not the end of the piece, Return to 6)

10) If end of piece, return to size values and increase them by the number of 03's added.

11) Save and move to next piece.

When all pieces are converted
12) Append all pieces back together

13) Reconstruct MDT file.

Obviously this probably still needs some more thought to it, but I think it's enough that I can code something together and then test it out and see how it does. Errors will probably still happen, but if this can help convert the bulk of the text with only a few errors that need hand checked and fixed I'd count that as a win here. If this can't be automated, the next option would be to do it all by hand which while doable, is very time consuming and opens itself up for it's own set of human errors.

One thing that could make this less complex and easier to implement though is if the default font can be set to the 8x16 font. This would eliminate the need for using the 0x03 control code to use that font and changing the size bytes. As a result this would then just become find the size byte, then byte swap the the necessary portion of the script piece.

[TrekkiesUnite118]

So time for some news.

First thanks to some help from Klarth/Mr.Conan over at SegaXtreme/Romhacking.net I've found out how to enable the 8x16 English font. The address 0x0601CEA4 has a byte that seems to control this. When it's set to x00 we get the 16x16 Japanese font. When it's set to x01 we get the ASCII 8x16 font.

Digging further in the debugger for YabaSanshiro I was able to find the assembly code that controls this. At 0x0601E2A4 there's these lines:

extu.b r0, r0
shlr2 r0
and #0x01, r0
mov.b r0, @r3

I've bolded the important code here. This code sets r0 to x00 and then writes it to the address at r3. That address happens to be 0x0601CEA4. Doing some testing, I was able to successfully force the game to use the 8x16 font by changing the code to this:

extu.b r0, r0
shlr2 r0
mov #0x01, r0
mov.b r0, @r3

This seems to do the trick. When viewing the memory address for this code I found these values:

600C 4009 C901 2300

This appears to align with the Opcodes for these instructions according to the SH-2 manual. If I'm understanding things correctly, I believe I need to change it to this:

600C 4009 E001 2300

Which would align with the Opcode for the replacement mov instruction. The problem is however, I can't seem to find this sequence of hex values anywhere in the game's executable, or anywhere in the games BIN/ISO for that matter. I found 2 values that were close, but they didn't seem to make any difference when I changed them. So I'm not sure what to do here.

Moving on however, I have some more news concerning this. I have continued work on my program for converting the files and I believe I may have successfully converted the PS1 script data over to the Saturn format. When combined with the memory edits above, I was able to get this:





Now this still isn't perfect. As while this cutscene plays fine, the game seems to hang afterwards when it tries to load Parm. The music and sound effects play, but the screen is black and the game doesn't seem to be progressing any further. Also not all the sound effects and voice clips seem to be playing during that first cutscene. So probably some debugging is needed to find out what's gone wrong. However the fact this seems to work in some cases looks promising that we can convert the English PS1 files over to the Saturn game rather easily.

I intend to release my code as I get it cleaned up as it's a bit sloppy at the moment. But if I get it working nicely, I'd imagine it could make it really easy for any future retranslation projects to apply their translation to both versions of the game.

[TrekkiesUnite118]

So I've been able to figure out how to get the game to use the 8x16 font by default. Turns out the reason I couldn't find the code that I needed to change is because it doesn't exist in the games compiled code. Instead those instructions are generated at run time based on values the game reads from other files. This could be a form of compression, or it could be some kind of automated code generation. I'm not really sure which.

However, I was able to find what value it's using to generate the code that determines what font to use. Turns out it's in the SCNRL.BIN file. At offset 0x22A4 is the value 0x9D. This value get's XOR'd with the value 0x54 to create 0xC9. This corresponds to the And instruction for SH-2. Changing this value to 0xB4 will instead result in the value 0xE0 being created, which is the Mov instruction we need instead to use the 8x16 Japanese font.

Now the game is still crashing after the intro cutscene when it tries to load Parm. However I think this is due to the MDT file having an issue with it, as when I swap back in the original Japanese one it no longer crashes. So I'm going to need to dig back into these files and figure out whats wrong.

[TrekkiesUnite118]

I removed the MDT file that was causing the game to crash and replaced it with it's Japanese counterpart for the time being while I figure out whats wrong. On the bright side though some of the other areas look to be working ok with some minor graphical glitches:



From the looks of it, in this area the palette data is wrong. So I probably need to figure out where it's trying to read from the wrong spot and fix it. The English PS1 version added/removed chunks of null data from the files, probably to increase/reduce padding for this very issue.

[TrekkiesUnite118]

So I'm still working on trying to figure out the Palette bug in some of the areas. It seems to be related to file size expansion, but it's kind of odd. Basically the offset in the MDT file header at 0x0D8 seems to be the entry that points to this data. And while the offset was updated in the file, it seems the game is only partially obeying it. Basically the game is still starting the read of the data from the correct spot in when it writes it into RAM. However it's stopping at what would be the originally stopping point for the original offset prior to the file size changing. The size of this data hasn't changed and it's still correct in the header. As a result of this, old data from the previous MDT file is still in RAM and is being copied over as Palette data, which isn't correct and is causing the partial graphical corruptions I'm seeing.

So I'm not entirely sure why this problem is happening. The only explanation is that it must be getting the value for where to stop reading from somewhere else. But where it's getting that info I haven't been able to track down yet.

In other news, I've put the code for the tools I've been writing up on GitHub:

https://github.com/TrekkiesUnite118/GrandiaTranslation

The tools are written in Java and hopefully should be easy to figure out. I'll be putting more info on the file formats as well as how to use the tools in the next few days when I have time. If anyone wants access as a contributor so they can add code/make changes let me know and I can add you.

[TrekkiesUnite118]

So it's been a while since I've posted an update, but I've finally solved the palette bug I'd been having with the automated conversion. Turns out the offsets in the MDT header 0x180 & 0x1F4 specify how many sectors need to be copied over from the CD for the file's data. This wasn't being updated so the game wasn't copying over all of the data. I've updated the tools to account for this and properly update the values when reconstructing the MDT files.

So with that out of the way I can move on to fixing some of the more minor issues that seem to be related to control codes. Basically audio isn't syncing as expected, and some of the text boxes aren't being positioned correctly or cleaning up correctly. I'd imagine this is probably some subtle difference between control codes across the Saturn and PS1.

Now, for something a bit more fun. I thought you guys might like to see some footage of the translated text in action. So here's a small bit I recorded from Mednafen:



Now to reiterate, this isn't perfect yet and there are bugs. So you'll get to see all of those as well.

[Bramsworth]

Surprised there's not much attention here. This is great work. My only complaint, and its unrelated to your work...

This line is so bothersome to look at. They could have gone any number of ways to avoid repeating the word "again" as the last word in each sentence. That's a definite no-no in localization. That said, from what I've seen in the most recent video it looks like most of the script is fine.

[Wesker]

Surprised there's not much attention here.

Oh, I'm definitely interested. It's just that unfortunately I'm lost with all the technical details. But a huge thanks goes to TrekkiesUnite118 for being one step closer of making one of the dreams of every Sega Saturn fan finally come true.

[Moirai]

Looking forward to one day playing this!

[TrekkiesUnite118]

So I was able to fix the Text Box bugs that were visible in the previous video. Basically for the smaller text boxes the game does a calculation based on the number of characters in it to determine it's size. The game was still acting like the font was 16x16 instead of 8x16 which was causing the boxes to be bigger than they should. This was resulting in some having an overflow and bugging out.

I did some digging and found that when the game read the 0x03 control code, it calls another subroutine for these text boxes that sets a value at the memory location of 0x06001ECC. For 16x16 font this value is set to 0x00, for 8x16 it should be set to 0x01. I was able to find where this value was coming from in the SCNRL.BIN file and make it so it permanently is set to 0x01 which has fixed the text boxes as you can see in the below picture:



With that out of the way I think I have the bulk of the issues ironed out with the main script. What remains to be done for the main script is the following:

1) Fix any other bugs that come about through play-testing (I'm expecting more to crop up that were similar to the fade-in issue with Parm).
2) Fix the audio synchronization issue.

The first will need to be done on a case by case basis as issues are discovered through play testing. The latter though is going to be tricky to figure out. One thing that really makes it hard is that emulators like Mednafen seem to draw faster than a real Saturn. So the audio synchronization is different between emulators and real hardware. On Real hardware the audio for the most part always plays, and the text severely lags behind in speed. So I'm thinking it may be an issue of text speed. I've reached out to the creator of the UnDub PS1 patch for help and they've been giving some suggestions to look at.

For now though I'm thinking of shifting focus onto the menus and what not as what remains with the main script isn't as pressing of an issue I'd say.

[Blades]

Keep up the great work.