removing the padding of every official rom

[chilled]

So this thread (started by myself) and the latest Sega Lord X video about the smallest Genesis games got me thinking about the size of the roms and if there is a reason to keep the padding on the games that don't use their full chip capacity (that is most of them, I guess, even if games like Gunstar Heroes have only 177 bytes of padding).

I've been reading here and there and couldn't find a reason to keep them in 2022 (SMD format is long dead, modern flashcarts support any rom size up to an upper limit, etc). It's mostly that people doesn't care about it, as it doesn't fix any problem in doing so. Space is not an issue anymore (except maybe mini consoles, etc. but they usually admit zipped roms which will be effectively the same size, as the padding compresses 99%). I guess it's only needed nowadays if you are burning the rom to a cart.

There would be some gains though: even faster loading speeds on flashcarts (this is critical for older versions like everdrive MD v3 which is probably still very popular) and sbc devices ala raspberry pi, lighter romsets and, most of all, knowing the real size of each game (I know it doesn't tell the full story as there are all kinds of optimizations and compression to varying degrees in each game).

How could I remove the padding automatically in a fullset ? Well, or just a file (from command line only) and I would create a batch to apply it to a whole folder. Can a hex editor be automated that way, or is there a specific tool as the ones for PSP and other disc-based games?

I think romsets can be discussed here as long as there are no links (or encouragement to download them) posted, so I'm wondering: why aren't there new romsets made with the padding removed ? At least on the more popular/downloaded sets (nes/snes I guess) ??

[ComradeOj]

I looked at some of my Genesis ROMs. It looks like they're padded out with hex $FF in most cases.

You could write a program what starts at the end of the ROM and works backwards removing any $FF bytes until it hits something that isn't $FF and moves on to the next file. It could be done in python, or maybe even something like a shell script or a batch file.

I don't know how well this would work, but it's an idea.

[roce]

The game can break that way, that's why it's not done. If you accidentally remove some bytes that are actual data, but happen to be 0/0xff/etc, it may crash the game or cause hard to find bugs.

[chilled]

but this has already been done to DS/3DS/GBA and PSP games (where the space savings are more apparent), there are tools for it. I cannot write any code myself except for simple batch commands after extensive googling.

EDIT I just quickly tested Alien Soldier: I removed the last 2 meg of padding and the game works perfectly (that would be in fact a 14 MEGA POWER game, not 16). But obviously if I remove intermediate padding the game doesn't even boot (I guess that changes where the data after that padding is stored, hence the pointers are all messed up). There are various regions of intermediate padding but they are minor compared to what is at the end of each game, which is the part I'd like to remove for every rom.
That would also require to mass checksumfix the new roms

Also it's not always "FF" the value used for padding: for Gargoyles it's "20". But a script that picks the last value of the rom and the proceeds to delete that one beginning from the end of the file up until it finds a different value, and then saves the result, would work. It this hard to write? Could someone do it in here if that doesn't take much time? Many thanks in advance.

I'm always bothered that Billytime's hacks (ability to save to sram, mostly) always increase the size of the rom about 8 megs each time, where 99% of that space is padding. I just trimmed Gargoyle's hack from 32 meg back to its original 24 this way. I guess he did just add his code AFTER the original rom padding instead of replacing the original padding with his code, something that fitted the rom space left, as proven with the trimming I just did.

EDIT2: been testing some roms manually, with and without checksum fixing (which doesn't seem to affect my mega everdrive x5 and is sometimes relevant for Kega Fusion if autochecksum fixing is disabled) and most roms do still work after trimming but I encountered a problem with Alien Soldier (european version), that does not boot on real hardware (it DOES on Kega Fusion) even if you only trim a small part of the padding at the end of the rom. Maybe there's some routine that checks for the filesize or points towards the end even if it's blank data, some sort of copy protection? For reference, the original untrimmed rom does boot on my Genesis, so it's not a region lock problem.

[Tony H]

EDIT2: been testing some roms manually, with and without checksum fixing (which doesn't seem to affect my mega everdrive x5 and is sometimes relevant for Kega Fusion if autochecksum fixing is disabled) and most roms do still work after trimming but I encountered a problem with Alien Soldier (european version), that does not boot on real hardware (it DOES on Kega Fusion) even if you only trim a small part of the padding at the end of the rom. Maybe there's some routine that checks for the filesize or points towards the end even if it's blank data, some sort of copy protection? For reference, the original untrimmed rom does boot on my Genesis, so it's not a region lock problem.

Just checked, and Alien Soldier (E) goes all the way to the very end of the ROM (0x200000) when doing a checksum check. This is most likely what is causing your problem. Patching this into your modified ROM should fix the problem 000468:6002. Go to ROM address 0x000468 and change the 6600 to 6002. This will force it to always pass the checksum check. If you have any others that won't boot, using a Game Genie "Master Code" for that game will usually fix the problem.

[chilled]

Just checked, and Alien Soldier (E) goes all the way to the very end of the ROM (0x200000) when doing a checksum check. This is most likely what is causing your problem. Patching this into your modified ROM should fix the problem 000468:6002. Go to ROM address 0x000468 and change the 6600 to 6002. This will force it to always pass the checksum check. If you have any others that won't boot, using a Game Genie "Master Code" for that game will usually fix the problem.

thanks, that worked!
The Japanese version doesn't check the whole content, and of course it is preferred (when using the [f1] which is not region locked anymore) because of the best start menu screen of the whole 16-bit generation.
I've been trimming my homebrews collection and also the hacks/translations when they increased the original rom size, mostly everything worked right after, except for some Sonic hacks which needed a checksum fix (many others still have incorrect checksum after the trimming but do work on the everdrive x5.