Fate/Stay Night [Realta Nua] PC version Mirror Moon TL insertion project

[NM64]

I see you did not answer regarding that vita version of "Into the night" - I'm guessing you simply don't know?

Basically the vita version of "Into the night" provided in the patch in the first post contains the custom "Into the night" that I put together from the OST, so I would greatly appreciate it if you could confirm/deny whether this "merge project" is using that single specific track I assembled rather than the iOS/Android copy (just doing a hashcheck between the files should reveal the answer).



Anyway, last thing - this "merge project" has the song "Whirlpool of Fate (Trial ver.)" listed and available in the built-in sound test music player thing, right? Right?

I unfortunately don't know where the ps2 loop data hides. I have some of them from waku's version but I have no idea if those were ripped or rather manually tuned.

Not sure why you're mentioning the PS2 loop data as I didn't ask anything about it, but that stuff is already present from PC-download Realta Nua, so wouldn't you just be re-using that...?

[TheUnknownIdentity]

I unfortunately don't know where the ps2 loop data hides. I have some of them from waku's version but I have no idea if those were ripped or rather manually tuned.

If it turns out they are either not in the audio files themselves or ffmpeg just can't find them I could probably track them down by stepping through the looping in the music menu. I already know where the function in the game for that menu is so it shouldn't be too difficult to track down.
... There are actually a couple ways I could do it. I may just hook the function called by @play and see what kind of values it returns. Either way I can eventually get you the loop info it's just a matter of how long it would take.

EDIT: It is present in the audio files ffmpeg just doesn't know how to get it I guess. I'll look into the file format and write something to make the corresponding .sli files.

The information needed is in the table here https://en.wikipedia.org/wiki/ADX_(f...t)#File_Header

The files are v4 format, should be able to create the files from the info in the header I just can't think of how atm since my brain isn't functioning.

[Quibi]

I'll use your version if you think it's better. And yes the trial version is there.
Since the whole point of having the PS2 tracks as well is because some of them are slightly different (I don't think I'll bundle all of them, only those that do sound a bit different), so I guess the loop files would be somewhat different.

Thanks Anonamous - I've managed to extract the loop data. It's a bit different as I suspected (unless I messed up - but it's close enough to expected values to seem reasonable - I used "Loop Begin Sample Index" and "Loop End Sample Index" for the start and end sample indices).

[NM64]

OK so, do you need me to get you a copy of that Type Moon Fes. torrent or something? There are a few out there, but I think there's only one that's as high quality of a rip as I alluded to, or maybe you still have it from back when (I think) I shared it?

If there's no seeders then say so - I'm currently not seeding it but I can fix that relatively easily (the data is stored on a different drive than what is in the actual PC that I seed things from).

Since the whole point of having the PS2 tracks as well is because some of them are slightly different (I don't think I'll bundle all of them, only those that do sound a bit different), so I guess the loop files would be somewhat different.

Wait, what? You're saying there's actual musical differences between PS2 Realta Nua and PC-download Realta Nua?

This is massive news to me as the only differences I knew of were the likes of the different looping in "Light and Darkness" whereby it doesn't actually start playing the second part until you reach a certain plot-point in the story...unless this is exactly the sort of thing you meant?

[TheUnknownIdentity]

This is massive news to me as the only differences I knew of were the likes of the different looping in "Light and Darkness" whereby it doesn't actually start playing the second part until you reach a certain plot-point in the story...unless this is exactly the sort of thing you meant?

There were audio tracks in the game titled 'demo0*.adx' being 1-4 but I do not know if it is these that are being referred to.

[Quibi]

The demo tracks are slight variations but not what I was talking about.
I don't remember there was a difference between the tracks, just that I made a "ps2" tab in the music menu in case there were.

And just to refresh my memory, what tracks besides "Turning Seasons" did RN change over the classic release?

[NM64]

And just to refresh my memory, what tracks besides "Turning Seasons" did RN change over the classic release?

Well it depends what you mean by "change" - do you mean that the actual musical composition for a specific song was altered, do you mean how its loops was changed, or that a different song actually plays during specific scenarios in the story?


Maybe I'm just being forgetful, but to be honest I don't remember "Turning Seasons" being changed during the actual story in Realta Nua but rather just a remixed version of the song being played during the credits that was specifically titled 'Realta Nua ver.' - case in point, the English patched RN PC-download contains both the original 'Turning Seasons' and the 'Realta Nua ver.' in its music test player thing...but I could be wrong here as I never did actually read through the entire VN in PC-DL Realta Nua, only in the original 2004 VN.

Nevertheless, I always just view that sort of music change for the credits as the same sort of thing as how it plays "Burst Up" during the Archer vs Lancer fight rather than "Clashing Souls" (which was one musical change I did not agree with) which would fall under the category of "a different song actually plays during specific scenarios in the story".

[SmilingWolf]

Just to confirm, we're not so OCD about image quality to bother combining the luma from the mobile WebP art assets with the chroma of the PC PNG art assets, correct?

To clarify, the mobile WebP art assets are encoded with 4:2:0 chroma sub-sampling and therefore have a luma (greyscale) resolution of 1280x960 but a chroma (color) resolution of 640x480. By contrast, the PC PNG art assets are encoded in full 4:4:4 chroma and therefore have a resolution of 800x600 for both the luma and chroma. This means to achieve the absolute best quality, you would take the chroma from the PNG, upscale it by 1.6x (perhaps via waifu2x), combine it with the luma from the WebP, and then save as a lossless format with 4:4:4 chroma.

You know what's good about an OCD-affected hobbyst with a Python installation?
He doesn't care how much time he will throw out the window in research and developement, as long as he gets what he thinks is the ultimate result.

I might be your man. That is, assuming this is still a thing and somebody is interested in the results.

So without further ado: ******************#!lkYQ2SrZ!LIiSG3Vyt...ayAksERZZ2pDLM
In the folder there is the script used for the merging and all the files used and resulting from the process.

Contents:
- mergelevels.py: does exactly what the name implies: takes two images in input, one from the android/ folder and the corresponding one from the waifu2x/ folder, extracts the luma plane from the first, the Cb and Cr planes from the second and merges them in a new image, saved in lossless WebP format in the merged/ folder;
- android: the webp images straight out of the APK, ripped personally;
- originals: the images used as source for waifu2x, straight outta Compton the Realta Nua PC edition xp3 archive;
- waifu2x: originals -> waifu2x with UpRGB model, TTA mode and 2x upsampling factor, because we want to decide how to downsample the final image instead of letting waifu2x do it with a bicubic filter, and we're going with lanczos because fuck yeah sharpness
- merged: what'chu think?

I have chosen 4 images that illustrate the different degrees of success this technique can have:
- 01汎用セイバー01左(決戦)_c.webp: no difference. Not that I can see at least, and I compare post-processed images on a large monitor in an obscured room so yeah, nothing to see here;
- iアインツロビー廃墟-(火).webp: little to no difference, the base of the handrail on the left, near the base of the stairs is a bit more vividly yellow-gold;
- a43a.webp: again little to no difference, the red light on the right side of the picture is redder/less blurry, Lancer's eyes look a little more alive, with the red a wee bit more vivid, the black outlines of the body suit look a bit more defined along his right harm. Not gonna hide I am a huge Lancer fag, so anything that makes him look better is a win to me. Also, notice how the finer details of the mesh of the fence have been lost in the waifu2x upsample, but have been preserved in the merge because we're using the HD luma plane;
- a43b.webp: the greatest success, you don't even have to stick your nose through the screen to notice the difference between the original, blurry as all hell version of the image and the layers-merged one. Either the VP8 compression did a number on this one, or they didn't have the HD master anymore and decided to upsample the 800x600 image.

Let's talk about filesize:
- the lossless, merged WebPs are, in average, 5 times as big as the lossy counterpart, BUT!
- they are, in average, only 30% larger than the 800x600 PNGs, with more than 2.5x the surface

The whole process can be automated. If you like what you see you give me a list of images you want processed, and in 3 days tops I'll have the results up.
If you have any questions or are up for further discussion I'll be here.

[Quibi]

@Nintendo - I'm talking about all kinds of changes. I think it was Jack who mentioned said changes so I'll ask him.

@SmilingWolf
Nice job on resurrecting an old idea from the grave, just when I thought work was done :P
Honestly since then I've automated the process of upscaling assets via waifu-2x caffe but I never did the chroma-luma thing (the upscaling was only done for uncensored CG anyway, so it wasn't necessary).

First - if I understood right - does x1.6 waifu2x upscaling really does x2 upscaling then downscales with bicubic? I didn't know that...
Also - I don't see where the lanczos downscaling is done in code?

I've been thinking... honestly all those "HD CG" the mobile has look pretty similar to just upscaling via waifu2x, although I do remember Nintendo compared the mobile assets vs upscaled and found that there're slight differences.
I wouldn't object to applying this algorithm to images - they really did screw up with that lancer picture.
It should be easy for standard 800x600 -> 1280x960 CG but there are many others, some of which are not upscaled exactly by 1.6.

Moreover, I originally had the HD assets from the 1.0.1 version of the mobile game.
I have the updated fate assets from 1.8.0 version (which fixed some images), but I could not find a way to acquire the UBW and HF assets from this later version.
So we'd probably need all the updated assets before applying this procedure.

Cool script btw

[SmilingWolf]

Waifu2x-Caffe is exactly what I use, so we pretty much have the same outputs.
Bicubic or area (I'm not familiar with this method and seems pretty much an OpenCV-only feature) downsampling in waifu2x-caffe: https://github.com/lltcggie/waifu2x-...Image.cpp#L758
Nagadomi himself confirmed, somewhere in the issues on GitHub, that the algorithm works ONLY for 2x, and anything different is a combination of iterated upsampling eventually followed by downsampling.
The lanczos downsampling is done in my script (line 35) since the images in the waifu2x folder are the original 2x upsampled without further processing.

I thought about images having slightly weird, non-exact-ratio scaling, so I'm downsampling the waifu'ed images to the exact sizes of the WebP originals. Should be good, unless I'm misunderstanding the problem.

You're telling me that saber.dat is not the only archive I should have, right? Should have thought so since I only have 2500-ish images and you mentioned more than 4k.
I'll have to hunt down the rest then.

The script is a (fairly complete) tech demo, I was planning to modify it to use the multiprocessing module to run the procedure for multiple images in parallel should you be interested.
Since I'm not familiar with that module I left that part out, but wouldn't take long to add... probably.

EDIT:
Yes, I can confirm most of these WebP must come from HD masters, at least the ones I checked. I'm pretty much a maniac for this kind of details, so I was positively impressed.
HOWEVER! The WebPs have one very predictable problem: very slight banding, unnoticeable unless you really are looking for reasons to complain, but it's there, especially in the scenese with a night sky.
The solution would be going all out and only using the upsampled PNGs, but after noticing the details of the rooftop mesh, I'm less than inclined to go this way. I'd rather keep the HD WebPs, and f*ck the banding

[Quibi]

I see. Nice. Seems like something we can batch process, shouldn't take too much effort.
Just need to get those updated assets...

Also, they really should add an option to choose the downsample algorithm

[SmilingWolf]

The script already handles batch processing.
Just follow the directory structure, and all the images in android/ will be matched by name to the ones in waifu2x/.
Batch processing was one of the core principles while writing the code to account for the large amount of assets, together with proper handling of unicode filenames and color conversion accuracy. 6 decimal digits and float64 values might be (are) overkill, but why not? It's slow already, will be good all the same lol

Estimated times:
4 images -> ~8 seconds, * (4000 images / 4) = ~2h 15 minutes for images upsampling
One single image -> ~3 seconds, * (4000 images / 8 threads if I get multiprocessing working) = ~25-30 minutes for planes merging

It'll probably take longer to upload than to process

Your call

[NM64]

In case anybody wonders where this random "SmilingWolf" person came from, see this thread.

For those that don't know, AV1 is a new fancy-pants open source video codec that's just been released, and Doom9 is pretty much the go-to place for any sort of technical discussion on all things digital video encoding/decoding and the like.


Anyway, I had a random thought last night - there's already quite a few versions of this visual novel that go by the title "Fate/Stay Night [Realta Nua]", so seeing as this merge project is quite a bit more deluxe than even the previous patch/es (which itself was more than just a translation patch), I'm wondering if we should add a subtitle so as to make it clear just what this is, like "Unofficial HD Version" or "Beast's Lair HD Edition", etc. Similarly, it would probably be a good idea for it to specifically say something like [BETA] in front or (BETA) at the end of the title, especially in the installer, just to make it extra clear what this is.

One reason for all of this is the idea that, for all we know, maybe Type-Moon will go and release and official HD version on PC at some point (Fate/stay night 20th anniversary? It's only 6 years away...). Additionally, calling it the "merged version" rather than "HD" may make people think of the original Realta Nua patch for the 2004 visual novel.

----------------------------------------------------------------

Hey wait a minute...if waifu2x only works at 2x internally which therefore means we need to downscale in order for the original upscaled 800x600 PNG art assets to match their 1280x960 WebP counterparts, wouldn't it then make more sense to just leave the 800x600 upscales at 1600x1200 and instead waifu2x the WebP images to match and then accordingly downscale to match?

Or would it be too much work at this point to change the VN's common native resolution from 1280x960 to 1600x1200?

And then of course there's the filesize considerations...AVIF can't come soon enough!

[Quibi]

I coded the HD support to be independent on the scale factor, so no problem there. But it could pose some other "issues".

Considering 1080p to be the prominent resolution nowadays (I think? I hope I'm not making this up), so if you play in windowed mode a height of 1200px would either overflow or we'd have to downsample it.
For full screen there's a "small" upscale (960 -> 1080) which I'd argue isn't too "bad" to warrant such upscaling.
The HD patch also makes the game a bit slower, and I think a larger resolution might make it worse.

I've currently named the next release "Fate/stay night Realta Nua Ultimate Edition" because it is certainly the ultimate way to experience the game :3
I doubt TM will re-release it again. There are already 435436546 versions lying around.
This name and the version of the patch are clearly mentioned in the installer (you can see that in the images I posted not long ago).
I just need to find some new images to use in the installer so it won't be a mirror of mirrormoon's installer (pun intended).

I actually recently discovered doom9 when I was researching a problem I had with MadVR (can't believe I used to watch anime without it, fancy upscaling are life changing).

Also, thanks for introducing us to SmilingWolf. Any help is appreciated. I could do the batch conversion as soon as I get my hands on the updated assets.

[SmilingWolf]

More help... or more work to do? Hah!

Got 'em, rin.dat and sakura.dat
Verified rin.dat just now, all good.
Considering I just pulled the dats from the official urls and used the fileinfos from 2015, I'd just go and assume the WebPs inside haven't changed in 3 years. A 1-byte change in offset would have resulted in catastrophic extraction failure, but nothing of the sort happened. I think whatever you've got already will be mighty fine

EDIT:
Verified sakura.dat too, all is good
By "verify" BTW I mean I opened every single image with IrfanView waiting for it error out in case of corrupted data. No error messages have been observed.

EDIT2:
Got hold of the HD rips of the OPs from the Fes BD Nintendo Maniac 64 was talking about. [ANK-Raws&TD] yes? It seems there's only one PS Vita OP, two PC and three PS2. That is, unless I'm tripping.
If worse comes to worse, maybe the original Vita ones can be used as-is or... waifu'ed maybe?

[Quibi]

Sounds too good to be true... I suspect those files might be earlier versions.

[SmilingWolf]

I pulled the urls right out of the fate::app::frame::payment::SubFramePayment::onUpda te() function inside libnative-activity.so, extracted from the latest APK.
Considering the names and the way the URL is built it looks legit, but it's quite late over here and my levels of concentration are dropping fast, so plenty of ways I might be wrong.

[Quibi]

I see, then it seems legit.
I'll just have to update my saber.dat then since that one I know was updated at 1.8.0, and it's now 1.9.6.

[SmilingWolf]

Done more tests, an horror story unravels: apparently sometimes they have reapplied the layers on the images differently between the PC and Android versions. It is the case, for example, of the 黒衣桜06e(近) image.
The effect layer has been applied with different levels of zoom between the two versions, making joining the different planes impossible because of the mismatches.
Hypothesis: this only affects character tiles (from fgimage.xp3), so we can still use this method for backgrounds. I have completed the multiprocess version of the script, so I can begin large scale testing.

[Quibi]

When I try to apply waifux2 with x2 and TTA - it takes about 2 minutes per file to complete, which is not deal for 4k images... (the TTA is the bottleneck, as without it takes only a couple of seconds).
If you have more processing power at your disposal I'd appreciate the help.

And yes, I think it's a good idea to apply this to the standard 800x600 -> 1280x960 backgrounds.