Fate/Stay Night [Realta Nua] PC version Mirror Moon TL insertion project

[Quibi]

So... merging is done. There's some more commented text I want to reinsert with the classic switch, but it's more of a lower priority now as the most important texts have been merged.

There's still a lot to do but the tedious part is done. Next on the list:
1) Fix HD support - there are a couple of broken scenes in HD, though other than that the game works as intended.
2) Divide the game to separate patches - while merging I also merged all the patches to make the workflow easier, but it's time to separate them once again.
3) Bug fixing.

The list doesn't end here but those are the top 3. When I've finished with the first two and fixed the more critical bugs I'll release a beta version.

[NM64]

Question - does the newer version of KiriKiri you're using support any platforms other than Windows? (Mac and Linux particularly)

Just thinking that, if it does and such platforms are within the scope of this "merge" project, then it'd probably be ideal for file size reasons to have all three version distributed in the same archive (that way the XP3 archives that make up the bulk of the data can be shared).

[Renari]

On the KiriKiriZ github it seems they are quite heavily using win32 libraries and Visual Studio. This means it's not likely that it runs on Mac or Linux.

[Quibi]

There's an active multi platform branch so that's something to look forward to, but probably not in the near future.

[Chlorium]

Having trouble with MONAKRKR font in kirikiroid2.
All spaces in choices and menu are "!", so, basically, it looks like "I'll!go!and!help!Sakura."
Some time ago it was with all the text, but now only with choices. Any way to fix it?

[Rokudaime]

2) Divide the game to separate patches - while merging I also merged all the patches to make the workflow easier, but it's time to separate them once again.

Why do you need to separate the patches again?

[Quibi]

So they will be optional.

[NM64]

So they will be optional.

I am confused why this matters for the "merge" project - aren't all the patches going to be distributed with the "merged VN" anyway, with each patch's functionality having some sort of option switch as well?

[XerBlade]

It has been my understanding that the plan for the merge project has always been to have the patches as options in the new installer. Just installing everything and only having the options as switches in the game sounds all well and good, but it can waste a lot of space for people, especially considering the size of some of these patches.

[NM64]

installer

Oh right, completely forgot that this will be a thing.

That makes me wonder - rather than compressing/packing the pre-compiled XP3 archives into the installer, couldn't the installer's file size be reduced by storing all the XP3 file contents as unpacked files and then assemble/compile those files into the necessary XP3 archives that the user has selected?

The idea is that a modern installer will compress data better than an XP3 archive will.

[Renari]

That's true, the issue would be that making an installer able to pack and unpack xp3 archives would require a fair amount of programming. If the file format is mapped out in its entirety (which I believe it is at this point, although I haven't looked myself), then it would require a few more hours of work. Although I'm pretty notorious for saying something should take me an hour or two to code and ends up taking an entire day working nonstop due to always running into issues I didn't anticipate.

[Dotelias]

Registered just to say thank you! can't wait for the next version, you guys are doing an awesome work

[NM64]

That's true, the issue would be that making an installer able to pack and unpack xp3 archives would require a fair amount of programming. If the file format is mapped out in its entirety (which I believe it is at this point, although I haven't looked myself), then it would require a few more hours of work. Although I'm pretty notorious for saying something should take me an hour or two to code and ends up taking an entire day working nonstop due to always running into issues I didn't anticipate.

It wouldn't have to unpack any XP3 archives, it'd only have to pack them. The only "extracting" the installer would have to do is from its own internal format.

Also I believe there's a command line option where you can specify if the XP3 archive is compressed or not, and using an uncompressed XP3 archive would be considerably quicker. Technically it may even be possible to have an installer option where the user can choose between "faster install but larger filesize" and "smaller filesize but slower install".

One way that could make the "compressed XP3 archive" option go faster would be to simply compress & pack as many XP3 archives at a time as your CPU has threads (I imagine that the XP3 compressor is only single-threaded). Therefore, on a 4-thread CPU, rather than waiting for a single XP3 archive to finish compressing before moving on to the next, it could instead run 4 instances of the XP3 compressor at a time (one per CPU thread) with little to no reduction in speed - this is extremely similar to how the "converter" in foobar2000 works.

If you want to get really fancy, you could even temporarily pack one of the smallest XP3 archives in an uncompressed XP3 and then a compressed XP3 behind-the-scenes to get an idea of the user's CPU power, and then use that to provide an estimated installation time based on the patches they've chosen and how many CPU cores & threads the user has (though, if the user doesn't have an SSD, the "uncompressed" installation may actually go faster if it just does one XP3 at a time).



Of course, all of this assumes that the priority is making the installer have the smallest filesize possible. If the priority is to instead have the fastest install possible, then the installer should be completely uncompressed and all of the XP3 archives should be pre-compiled, therefore making the actual installation purely I/O limited so you'd want to make sure that the installer only copies a single file at a time to minimize I/O bottlenecking as much as possible (even on an SSD, the sequential reads/writes are faster than random read/writes).

[Renari]

Right, for some reason I thought you meant having the installer repack the replacements in the original xp3. However I see little reason to go through the trouble of having the installer pack the xp3, rather than just compressing the xp3 itself. The difference in filesize should be marginal.

[Quibi]

The idea I had for the installer is to merge the existing xp3 files from the user's PC during install (using kikiriki to unpack, merge, and then repack).
It's a bit time consuming but it saves up space by reducing duplicate files. I guess it will also be easy to just use all the xp3 files from all routes as is, which will reduce install time but increase disk size.

That's just for the existing data. For the patches themselves the only "problematic" ones are the OP patch and the HD assets, each weighing about ~0.5GB.
Since the movies are already compressed by their encoding, and stored in an uncompressed xp3 file, I don't see how we can further optimize this.
Not sure how much effect xp3 compression has on the HD assets, I'll have to test it.

[NM64]

The difference in filesize should be marginal.

That really depends on what compression algorithms are used and how well the contents can be compressed; algorithm-wise, there can be a considerable difference in compression ratio between something like DEFLATE (default for ZIP archives) and LZMA2 (default for 7z archives).


Ideally one should make four quick & dirty test installers and compare the filesize of each installer:

#1. [slowest install] compressed installer with the file contents of a given XP3 and the necessary command-line tool(s) that'd be required to re-pack the file contents back into an XP3 archive

#2. compressed installer with a compressed XP3

#3. compressed installer with an uncompressed XP3 (may actually compress better than #2 without being slower)

#4. [fastest install] uncompressed installer with a compressed XP3


----------------------------------------------------------------

That's just for the existing data. For the patches themselves the only "problematic" ones are the OP patch and the HD assets, each weighing about ~0.5GB.
Since the movies are already compressed by their encoding, and stored in an uncompressed xp3 file, I don't see how we can further optimize this.
Not sure how much effect xp3 compression has on the HD assets, I'll have to test it.

This reminds me - are the video files in the merge project's OP patch the exact same video files in the current OP patch, or did people finally replace them with the higher-quality ones that I mentioned like a year or two ago? (some even have a smaller filesize while still being higher quality)

[Quibi]

I still have no idea which ones I'll use. There're the ones we got from the mobile (ogv, ~30mb), vita (mp4, ~90mb), and some other ones you found.
I'm inclined towards the vita ones but we'll have to investigate this as well...

e: those are just the vita ones. I'm not even sure what's the source of the ps2 videos.

[NM64]

NOTE: I made a pretty large edit to the bits about the 2004 OPs, so I felt it was necessary to delete and repost this.




The Vita versions are almost certainly better quality; ogv would be theora which would be similar to xvid, while Vita might be xvid at worst but even then would have 3x the bitrate.

Of course, the Vita fate route OP was on the Type-Moon Fes blu-ray so that is clearly the best source for that; video format shouldn't require any re-encoding unless you feel that the bitrate is too insane (audio may require re-encoding though).

PS2 is also on the Type-Moon Fes blu-ray, but they need cropping, sharpness "enhancement", and contrast adjustment due to poor mastering or something (I mean the videos are even window-boxed). Again, the audio may require re-encoding.

For 2004 PC OPs, the Type Moon Fes blu-ray once again seems to be the best quality, but things aren't so simple - the original videos were 29.97fps, but there are blu-ray rips floating around that have been telecined to 24fps (which is what the PS2 and Vita/mobile OPs use). Therefore, a more accurate statement would be that the Type-Moon Fes blu-ray is the best quality, but only if the blu-ray rip preserved the original 29.97fps frame rate.


Technically for most of these, we could even source the lossless audio from an OST and then transcode that to AAC (or even Opus if mkv containers will work) and get higher-quality audio with a lower bitrate; some trimming/padding of the audio waveform may be necessary however in order to make it align correctly with the video.

[Quibi]

I see. So setting aside post-processing, we take whatever we can from the Type Moon Fes blu-ray, while ubw&hf vita op from the vita.
Not sure I understand what you meant about the 2004 videos - if the rips aren't 29.97fps then what would be the superior source?

[NM64]

You know, maybe if we wait about 3 years, they'll release a Type Moon Fes 20th Anniversary Blu-ray with all the Vita/mobile OP videos!

Also psst Quibi, I replied to your PM five days ago but you never replied back...

ubw&hf vita op from the vita

Question - are the mobile OP videos higher resolution than the Vita? While I doubt that any higher resolution will make up for the bitrate deficit (2.5mbps 720p Theora should utterly pale in comparison to 7.5mbps 544p xvid, especially if the vita actually used h.264), it would probably still be a good idea to at least take a look.

I mean, if the quality difference is as big as I reckon it is, then it should be obvious even by just eye-balling it. For reference, MPC-HC, mpv, and VLC should be able to play the videos out-of-the-box, and if the Vita videos are h.264 then you can likely even play them directly in your web browser if it isn't Safari, IE, or Edge.


Heck, if the video quality of the Vita OPs are as good as they sound like they'd be, then one could possibly even extract all of the frames to individual images and apply a waifu2x upscaling to get a result that looks better than the mobile OPs in quality and resolution.

if the rips aren't 29.97fps

Don't worry, at least one rip exists at 29.97fps since I had a copy of it already (though it was only 720p; not sure if a 1080p version exists at 29.97fps).