Alright, as announced in the last post I’d like to give you a brief overview of the recent changes.

First of all, Lover Able is built with ‘Yuka System’, quite an old Japanese visual novel engine with barely any information available online.
Because of this, we originally didn’t even know about some of its features, which were only discovered after countless hours spent examining its machine instructions.

There are for example certain control codes you can insert into lines of text, for example @r(n,x) to insert furigana centered above the the previous n characters.
One of these control sequences is @m(n), which scans the next n characters and inserts a line break if they would overflow the current line.
Since I’ve written a custom compiler for Yuka scripts, I was previously able to add a workaround for word wrapping by inserting the appropriate amount of spaces to push a word to the next line.
This was however very inflexible as it had to be performed at compile time, which meant that variable strings like the player name could not be wrapped correctly.
Discovering this hidden feature allowed me to simply prepend every word by a @m clause with the length of the word.
There were still some issues though, were the engine would wrap the last character of a word even though it would have fit in the current line.
The technical reason for this was that @m clauses calculate the actual length of the string in pixels, whereas the code responsible to actually add the characters to the text box assumed every char to be a full width character.
My (slightly hacky) fix for this was to simply treat every character as half width, which is more appropriate for english text.

The second feature I tackled was support for proportional fonts.
Japanese text layout is notoriously simple; Every character has the same width (or half of it), which is why many Japanese visual novel engines don’t contain any sophisticated text layout algorithms. This however results in the typical “blocky” look since every character (even m and i) will take up exactly the same amount of space and is also why VNs usually look terrible when used with a non-monospaced font.
Adding makeshift support for proportional fonts (which means that each character has an individual width) was surprisingly simple since the engine already calculates the width of each character to determine the size of the render buffer it needs to allocate. So it was just a matter of accessing this information and advancing the character position by the specified amount. I ended up subtracting 3 pixels because the render buffer always contains a slight padding on either side.
I might revisit this later and make it a bit more robust (it currently looks a bit off when you disable the text outline since the characters end up narrower).

The last (and by far most complex) change was support for unicode characters.
Japanese software often utilizes the infamous Shift-JIS character encoding to represent text.
In Shift-JIS there are single and double byte characters, which btw is the way the engine used to tell apart half width from full width characters.
Unfortunately it doesn’t have a mapping for many characters (like the é in café or certain punctuation characters), since they are not needed in Japanese text.
Therefore, it used to be impossible to display these characters in the game and we had to use alternatives (like e instead of é).
Before I’ll start to explain the workaround (which again is a very makeshift solution ^^), I should probably mention some of the methods and classes that are involved in text output within Yuka.

Note that this is the result of literal months of analyzing thousands of machine instructions. It’s the first time I’ve really modified an existing program without access to the source code, so natually I still had to get used to the tools and develop my own workflow. All classes and methods were named by me, since I have no clue what their original names were.

Classes:

'YukaGraphic'
This class represents an image and contains pointers to several pixel buffers as well as bitmap handles and general information about the image (like its dimensions).
YukaGraphics can have multiple ordinal states, each with multiple frames of animation.

'YukaSprite'
These massive, almighty structures are used to represent every interactive object on screen. This includes buttons and – most importantly – text boxes.
Basically everything that isn’t just a still image has to be a sprite.
Every sprite contains multiple graphics, a position, slot ID, text information and a lot more I didn’t identify yet.
“Text information” includes style (color, font, text effects), the position of the next character, bounds of the text area and a buffer allowing for up to 1000 text elements.
In total, each sprite object occupies almost 5 kilobytes of memory (excluding additionally allocated buffers).

'TextElement'
This class represents anything that might be added to a text box. As far as I know there are 3 modes for these: character mode, string mode and icon mode.
So a text element can either display a single char (like for the main text, which appears one character at a time), an entire string of text (used on choice screens), or an arbitrary (possibly animated) image.
Each text element contains information about its type, position, dimensions, visibility, text style, content and some more stuff (like animation state for icons).

There are many more classes, but these should be enough to understand this particular workaround.

Functions:

YukaScriptValue* __cdecl rtStrOut(YukaScriptContext* ctx, int argc, void* argv[]);
This is the script method responsible for outputting a line to the text box character by character.
I chose the rt prefix to mark functions which can be called by Yuka scripts at runtime, they’re basically API functions.

int __cdecl ProcessCharacterOrControlCode(YukaSprite* pTextbox, char* pString, YukaScriptContext* pContext);
Put simply, this function looks at the next character in the string pointed to by pString, determines whether it is a control code or a regular character and reacts accordingly, calling appropriate Handle_X methods for control codes or adding the character to the text box.
It returns the number of consumed bytes from the input string.

int __thiscall YukaSprite::AppendCharacterWithCurrentStyle(YukaSprite* this, char* pChar);
A wrapper that calls YukaSprite::AppendCharacter with the current text style of the sprite.

bool __thiscall YukaSprite::AppendCharacter(YukaSprite* this, char* pChar, int fontIndex, int textColor, int fontEffect, int textShadowColor, int textBorderColor);
This method allocates a new text element and initializes it by calling TextElement::CreateFromChar, then adds it to the sprite’s text element buffer (YukaSprite::AppendTextElement).
This is also the method I modified earlier because it advances the “cursor position” after adding a character.

bool __thiscall YukaSprite::AppendTextElement(YukaSprite* this, TextElement* textElem);
Simply adds the text element to the sprite’s internal buffer.

bool __thiscall TextElement::CreateFromChar(TextElement* this, char* pChar, int x, int y, int fontIndex, int textColor, int textEffect, int textShadowColor, int textBorderColor);
This method is basically a constructor of the TextElement class, which initializes a text element in “single character mode”.

bool __cdecl RenderTextElement(int fontIndex, TextElement* textElem);
Basically just a wrapper function to load a font by its index and call FontInfo::RenderTextElement on it.

int __thiscall FontInfo::RenderTextElement(FontInfo* this, TextElement* textElem);
Another wrapper that retrieves the single character from a text element and passes it as a string to FontInfo::RenderString

bool __thiscall FontInfo::RenderString(FontInfo* this, TextElement* textElem, char* string);
This massive function is where the actual text rendering takes place.

BOOL TextOutA(_In_ HDC hdc, _In_ int nXStart, _In_ int nYStart, _In_ LPCSTR lpString, _In_ int cchString);
This is a Windows API function (hence the different signature), used to render an ansi string to an image.

BOOL TextOutW(_In_ HDC hdc, _In_ int nXStart, _In_ int nYStart, _In_ LPCWSTR lpString, _In_ int cchString);
Also a WinAPI function, but this one expects the passed string to be Utf-16 encoded.

Again, there are far more than these, but they are most important for this workaround.

I’ve spent quite some time talking with frc_, a very experienced programmer and reverse-engineer (is that a word?), tossing around ideas on how to implement such a feature (he has done a similar thing for the CatSystem2 engine).
So what approach did I end up taking?
The basic idea is to add a new control code @u(xxxx) to insert arbitrary Utf-16 codepoints as characters.

First of all, I needed a way to render Utf-16 characters at all. Since Yuka doesn’t use it, there is no import of the TextOutW function you would need for that.
By adding a jump instruction to the start of the FontInfo::RenderString function, I redirected the control flow to execute my custom logic instead of the regular function.
And what does it do?

First, it checks whether it was executed before. If it’s the first time, it loads the Windows library “Gdi32.dll”, which contains the “graphics device interface” methods, aka everything graphics related. Next, it retreives the address within that dll, where the function “TextOutW” is defined and stores it in a reserved memory location I’ll refer to as _TextOutW. It also copies the address of the imported function TextOutA to another location labeled _TextOutA.
After this initial setup step, it checks whether the text element that is supposed to be rendered contains a Utf-16 codepoint (more on that later). If that is the case, it overwrites the imported TextOutA function with the previously initialized pointer _TextOutW.
Then, another jump instruction tells the program to continue the regular RenderString method.

At the end of that method, another simple hook restores the TextOutA function to the address we previously saved as _TextOutA.

This actually works, but we’re far from finished as there is currently no way to actually create a text element that contains a Utf-16 code point.
And that’s exactly what the next step is about:

Similar to the TextElement::CreateFromChar method, I wrote a new one called TextElement::CreateFromUtf16, which gets passed a code point as integer instead of a char pointer.
This method is very straightforward, as it simply copies the passed method parameters to the appropriate fields within the text element.
Now, I still needed a way to tell apart these Utf-16 text elements from regular character elements. I did that by utilizing an unused byte within the TextElement structure.

Now to the most difficult part: How do I decide, when such a text element should be created?
That’s where the actual @u(xxxx) control code from earlier comes into play.

As I mentioned, the ProcessControlCodeOrCharacter method contains logic to look for certain control codes like @m, @r and some others.
So what I did was to add another hook that checks for @u and calls another new function called StrOut_HandleUnicode.

This is probably the most complex method I’ve ever written in plain assembler.
It copies the 4 character hex string within the parentheses into a buffer, calls strtol to get the numeric value (the Utf-16 code point), allocates and initializes a new TextElement, calls TextElement::CreateFromUtf16 on it, adds it to the sprite and does all sorts of additional setup required to display the character.

Now all that was missing was to rewrite my YukaScript compiler to replace characters, that can’t be encoded in Shift-JIS, by the appropriate @u control code.

And that’s about it, 3 days and 3 nights of pure assembler madness 😀
I hope this little insight into my work was a bit interesting and it will improve the quality of the final game 🙂

– Atom