Back in the days of SDK2, VRChat worlds rarely held more substance than a glorified museum of Unity assets. Creativity as far as model and material work was effectively limitless, but when it came to interactivity, well, you had the Trigger-Action system, but this could only really handle the most basic of interactivity. All complex mechanics relied on either generating complicated networks of Triggers within the editor, or using other Unity features such as shaders or Animator FSMs for the bulk of their processing.

But then enters a new challenger in the ring: Networking.

You see, VRChat is a social game. While it isn’t too uncommon for people to gad around worlds on their own, by and large, building worlds is largely an exercise in trying to make an experience engaging for a group. If everyone’s seeing different things, then you force players to choose between their interest in your world and their interest in their social group, and 9 times out of 10, the group wins.

Suffice to say, it’s usually within our interest to give networking a serious thought. In SDK2, our options are:

SDK2 Networking Option 1: Don’t

For many worlds, this is the only option they go for. It might seem kind of like cheating to include this, but as VRChat’s components build networking into them by default, having your objects not engage in networking is actually a choice you have to make.

Within this category falls the classic ‘preference’ boards you see in the majority of “relax” or “hangout” worlds. These options let you choose things like toggling or adjusting background music, colliders, mirrors, quality settings, and more.

It makes sense why you’d want every player to handle these individually, but I tie their frequency not just to a want for customizability, but also to the fact that they’re one of the easiest pieces of interactivity to add, given there is no expectation for them to be networked at all.

SDK2 Networking Option 2: Transform sync

This is the most basic form of networking in SDK2. A VRC_ObjectSync sends the Transform information (that is, the position, the rotation, and the scale) of the object it is attached to from the Master Client (i.e., the player who has been in the world the longest) to the rest of the instance.

As this form of networking is effectively drag-and-drop, it’s not uncommon to see at least one or two synced pickups in an SDK2 world. They’re great for little props and decorations. But they’re not very useful for functional logic. For that, we need to step up to trigger sync.

SDK2 Networking Option 3: Trigger sync

Finally, we get to Triggers. The Trigger-Actions system ties a Trigger (such as OnEnterCollider) with an Action (such as PlayAnimation).

When chosen to be synced, they can have one of three sync types:

• Unbuffered: The trigger’s activation is only synced to those currently in the session. People who join later ‘miss’ the event.
• Once: The last activation of a given trigger is synced to new users, with events being replayed in order of the last time they were executed. For instance, if a player presses Open Door, Close Door, and then Open Door, new players who join will receive Close Door and Open Door in order, leaving the door correctly open, but skipping all previous invocations of either trigger.
• Buffered: This mode makes it so that every time the trigger is activated, it is added to a continuously growing backlog of invocations, all of which get replayed for new players that join. This mode is pretty inefficient, and so it is not really recommended, but it leads to relatively consistent results.

This is as complicated as the interactivity and networking really gets in SDK2. All networking statefulness is represented by the history of Trigger invocations within the world.

Enter Udon

After a few months of Alpha, SDK3W and Udon was released for VRChat on April 2nd, 2020. It offers a complete replacement to the Trigger-Actions system in the form of UdonBehaviours - an analogue to the MonoBehaviours of standard Unity, and a node editor to create these behaviours.

This node tree compiles to Udon Assembly, an assembly language that is actually executed by the runtime. UdonBehaviours can take Udon Assembly directly, meaning other tools can produce this assembly by transpiling from other languages.

.data_start

    .sync Held, linear
    
    __Boolean_0: %SystemBoolean, null
    __instance_0: %UnityEngineTransform, this
    __value_0: %UnityEngineQuaternion, null
    __a_0: %UnityEngineQuaternion, null
    __b_0: %UnityEngineQuaternion, null
    __t_0: %SystemSingle, null
    __Quaternion_0: %UnityEngineQuaternion, null
    __Quaternion_1: %UnityEngineQuaternion, null
    __fromDirection_0: %UnityEngineVector3, null
    __toDirection_0: %UnityEngineVector3, null
    __instance_2: %UnityEngineTransform, this
    __instance_1: %UnityEngineTransform, this
    __Single_0: %SystemSingle, null
    __Single_1: %SystemSingle, null
    __Boolean_1: %SystemBoolean, null
    __Boolean_2: %SystemBoolean, null
    Held: %SystemBoolean, null

.data_end

.code_start

    .export _update
    
    _update:
    
        PUSH, Held
        JUMP_IF_FALSE, 0x00000018
        JUMP, 0x0000011C
        PUSH, __instance_1
        PUSH, __a_0
        EXTERN, "UnityEngineTransform.__get_rotation__UnityEngineQuaternion"
        PUSH, __instance_2
        PUSH, __fromDirection_0
        EXTERN, "UnityEngineTransform.__get_up__UnityEngineVector3"
        PUSH, __toDirection_0
        EXTERN, "UnityEngineVector3.__get_up__UnityEngineVector3"
        PUSH, __fromDirection_0
        PUSH, __toDirection_0
        PUSH, __Quaternion_0
        EXTERN, "UnityEngineQuaternion.__FromToRotation__UnityEngineVector3_UnityEngineVector3__UnityEngineQuaternion"
        PUSH, __a_0
        PUSH, __Quaternion_1
        COPY
        PUSH, __Quaternion_0
        PUSH, __a_0
        PUSH, __b_0
        EXTERN, "UnityEngineQuaternion.__op_Multiply__UnityEngineQuaternion_UnityEngineQuaternion__UnityEngineQuaternion"
        PUSH, __Single_0
        EXTERN, "UnityEngineTime.__get_deltaTime__SystemSingle"
        PUSH, __Single_0
        PUSH, __Single_1
        PUSH, __t_0
        EXTERN, "SystemSingle.__op_Multiplication__SystemSingle_SystemSingle__SystemSingle"
        PUSH, __a_0
        PUSH, __b_0
        PUSH, __t_0
        PUSH, __value_0
        EXTERN, "UnityEngineQuaternion.__Slerp__UnityEngineQuaternion_UnityEngineQuaternion_SystemSingle__UnityEngineQuaternion"
        PUSH, __instance_0
        PUSH, __value_0
        EXTERN, "UnityEngineTransform.__set_rotation__UnityEngineQuaternion__SystemVoid"
        JUMP, 0xFFFFFFFC
    
    .export _onPickup
    
    _onPickup:
    
        PUSH, __Boolean_1
        PUSH, Held
        COPY
        JUMP, 0xFFFFFFFC
    
    .export _onDrop
    
    _onDrop:
    
        PUSH, __Boolean_2
        PUSH, Held
        COPY
        JUMP, 0xFFFFFFFC
    

.code_end

This is a huge step up from the Trigger-Actions system. Along with the power of UdonSharp, we can write code that feels exactly like it works in Unity.

using UnityEngine;
using UdonSharp;

public class RotatingCubeBehaviour : UdonSharpBehaviour
{
    private void Update()
    {
        transform.Rotate(Vector3.up, 90f * Time.deltaTime);
    }
}

If you’re like me, a step-up of this scale should sound exciting. Really exciting. And excited I was, upon the release of Udon. With a Turing-complete language on our backs, surely we’d be able to implement the most complicated of game engine systems, all within the closed box of VRChat. I even implemented a QR code generator and a cryptographic hashing library, so surely we’d see a huge shift in the vastness and complexity of VRChat worlds!

Well, about that…

Networking with Udon

Despite the vast possibilities with Udon, we still don’t really often see many complex systems implemented in it. The scenery on this has been changing recently, as more people attempt to dip their feet in it, but I’m wagering the reason why we haven’t been seeing much is because networking is still really hard to get working.

Early on in SDK3 networking, we had two options for sync:

SDK3 Networking Option 1: Events

This method echoes the Trigger sync method from SDK2. It lets you invoke a method on an object for everyone in an instance, or just the Owner of the object.

public void SomeEvent() {
    ...
}

public void OnInteract() {
    SendCustomNetworkEvent(NetworkEventTarget.All, "SomeEvent");
}

Unlike Triggers, events never reply for late joiners. You’re either there or you miss it. In addition, like Triggers, there is no extra statefulness other than the name of the event itself. For something to be triggered in this way, a public method must be defined with the corresponding name.

SDK3 Networking Option 2: Continuous sync fields

This method echoes the Transform sync mode from SDK2. It’s a bit more flexible, because you can sync any field you want, and specify the kind of interpolation to be used.

[UdonBehaviourSyncMode(BehaviourSyncMode.Continuous)]
public class Example : UdonSharpBehaviour 
{ 
    [UdonSynced(UdonSyncMode.Smooth)]
    public float synchronizedFloat;
}

The owner of the object is the one who ‘sends’ their value for the field. Everyone else in the session ‘receives’ the value. This comes with an obvious caveat: to send data to others, you need to own the object that holds the data.

It’s particularly inhibiting for two-way communication, if user A wants to send data to user B, and then user B wants to send something back, either there must be two objects, one owned by each, or user B needs to ‘take ownership’ of the object from user A, which is slow and requires a network negotiation before it will even work.

Our problem here scales with user count, as well. If your world has a hard user cap of 80 users, you’re either going to have users constantly fighting for ownership of the object, or you need to have at least 80 objects in your scene, and assign one to each user. Unsurprisingly, most worlds end up going the later route.

There’s one more sync mode these days, though, and it tries to bridge the gap between both events and continuous sync, with… some success.

SDK3 Networking Option 3: Manual sync fields

This mode has the statefulness of field sync, but with the ‘on-demand’ nature of events (almost). The way it works is, you define your behaviour as manually synced, and then you can update fields and ‘prompt’ that update to be sent.

[UdonBehaviourSyncMode(BehaviourSyncMode.Manual)]
public class Example : UdonSharpBehaviour 
{ 
    // Note that this means no interpolation, not that it isn't synced.
    [UdonSynced(UdonSyncMode.None)]
    public float synchronizedFloat;

    public void OnInteract() {
        synchronizedFloat += 1.0f;
        RequestSerialization();
    }
}

This boasts a better approach for a lot of typical stateful network actions than the other sync modes, and, it comes with a guarantee that Manual syncs should be ‘guaranteed’, whereas Continuous syncs may drop updates occasionally.

This mode is also lower latency and capable of updating more frequently:

put together a better test video to show people.
pink box is manual sync, yellow box is continuous sync. recorded both windows so people could see the delay. #VRChat #madewithudon pic.twitter.com/Q6FkBmK6Gj

— JetDogLabs (@jetdog8808_dev) May 19, 2021

There is still the disadvantage of ownership, though, so it’s still not a perfect solution.

Which brings me on to the topic that made me write all of this in the first place:

Fun experiments in Udon Networking Land

One of the things I love to do is write tools. It should be no surprise that better tools save time, but I find that good tooling is also one of the best ways to inspire creativity.

For this reason, I’ve been trying for a long time to devise good tools and frameworks for Udon. Thanks to Udon Assembly, and the power of Unity editor scripting, I can try and design frameworks that can solve many of the harrowing networking problems Udon comes with, and enable easy, intuitive ways to make syncable experiences in VRChat, using technology we already use for making great experiences in full games.

This comes with the weight of having to try and design optimal strategies for certain functionalities to be implemented. We don’t have the same kind of hands-on access to the networking pipelines we would in a regular game, so we have to make our choices carefully.

With this in mind, there are a couple tests I wanted to try and see the effectiveness and impact of, both in terms of performance and design.

Experiment 1: Creating an ungodly amount of public methods

In this experiment, we’re going to generate a bunch of text on one client, and try to sync it to everyone else by rapid firing events. To do this, I’ve decided I’m going to implement a method for every printable two-character pair in ASCII, for a total of 9,312 methods (inclusive range of [32-127] for both, plus a 0 for the latter, so it can do single character sends at the end of strings).

The text will be communicated along by sending the first two characters, waiting 0.05 seconds, sending the next, and so on, until the text is complete.

I was originally going to do this with the full [0-255] range on both, but UdonSharp would take 6 minutes to process the file every time I touched the editor, so I cut it down a bit. With the resulting file at a firm 55 seconds to process, let’s give it a try:

I’m going to be really honest, I was expecting this to go shockingly badly. Not only is the update rate pretty fast, but I had heard stuff about how a high public method count can affect Udon performance. Guess that didn’t end up being true.

My main interest regarding doing this was seeing how practical it was to expose labels in an orchestration system to other more Unity-based triggers. For instance, when executing an AI stage script, the AI needs to wait for a certain event to occur, but the event itself is dependent on an animation.

Animation events can trigger functions, but from the animation’s point of view, it only sees the UdonBehaviour, not the functions we define. We can use UdonBehaviour.SendCustomEvent, but that can only take a function name and no parameters. I wanted to know if simply making a function generate for all trigger labels in the stage script would affect performance, and it doesn’t seem like it does. It did affect world size, but not much more than the actual size of the text for each function name.

The one big disadvantage here is that you still can’t use it for arbitrary data. You can only use it for enumerable data, where you know every possible outcome.

So let’s try something else interesting. We’re going to have to sacrifice the no-ownership-required, though.

Experiment 2: Serializing a lot

This is going to be our next experiment: sending data by serializing a lot. We’ll be using the manual sync mode, loading a small buffer of 2 characters (just like the previous experiment), and then serializing.

Unlike the last experiment, where we space the events out evenly, we’re instead going to try and use OnPostSerialization to send the events as fast as humanly possible.

Our advantage here is that the buffer can contain whatever we want in theory. I’ll also try scaling it up to larger character sequences as we go along. All right, let’s give it a try:

using UdonSharp;
using UnityEngine;
using VRC.SDKBase;
using TMPro;

public class RapidSerialize : UdonSharpBehaviour
{
    #region Filled in via the Inspector
    public string sourceText;
    public TextMeshProUGUI output;
    #endregion

    // The total text, per client
    private string fullBufferText;

    // The buffer text of X amount of characters
    [UdonSynced, FieldChangeCallback(nameof(BufferText))]
    private string bufferText;
    private string BufferText
    {
        set
        {
            bufferText = value;
            if (value == "")
                fullBufferText = "";
            else
                fullBufferText += value;
        }
        get => bufferText;
    }

    // Whether serialization is done or not
    private bool serializeReady = true;
    // A timer, so that we can provide a timeout for serialization
    private float timer = 0.0f;
    // Where in the source text we're buffering from
    private int index = 0;

    private void Update()
    {
        output.text = string.Format("{0:F2}\n{1}", 1.0f / Time.deltaTime, fullBufferText);

        if (!Networking.IsOwner(gameObject)) return;

        timer += Time.deltaTime;

        // Run either when serialization completes, or when 5 seconds has passed (fallback)
        // Note that with only one player in the session, serialization does not happen at all,
        //  so each chunk will always take 5 seconds.
        // Maybe write in an exception case for this?
        if (serializeReady || timer >= 5.0f)
        {
            timer = 0.0f;
            serializeReady = false;

            if (index >= sourceText.Length)
            {
                // Reset once the text is done
                BufferText = "";
                index = 0;
            }
            else
            {
                // Change this 2 to anything else to increase the amount of data buffered
                BufferText = sourceText.Substring(index, Mathf.Min(2, sourceText.Length - index));
                index += 2;
            }
            RequestSerialization();
        }
    }

    public override void OnPostSerialization(VRC.Udon.Common.SerializationResult result)
    {
        serializeReady = true;
    }
}

The biggest thing that I took away from this, is that OnPostSerialization doesn’t run if you’re the only person in the session. I had somewhat expected this, but it was nice to get confirmation on it, so if you have any logic that relies on timing serialization, you’ll need to make sure you have some exception for when the player is alone.

Other than that, the speed seemed to be pretty similar to the event method. However, scaling the synced data up to 25 characters didn’t make any difference to the speed of update, so more data can be shipped. It is worth noting that Udon intentionally does have a limit to how much data can sync per behaviour though, so you can’t just ship huge payloads all at once.

My main interest with this method was syncing map data. I had an idea for a procedurally generated map in Udon, and I needed it to be possible to send data to other clients to reproduce the map. This does actually seem pretty usable, and while we run into the owner limitation again, it’s unlikely you’ll be mass sending something like map data from someone who isn’t the owner, so this does seem pretty suitable.

A final word: ‘drop-outs’

The one final big hurdle to networking in Udon is network drop-out. Sometimes this happens gracefully (like when the owner leaves), and sometimes it happens abruptly (like when the owner crashes).

A lot of networking-heavy worlds end up mentioning at their start that the master leaving will break the world. Some worlds can handle the master leaving, but can’t handle the master crashing. For my framework and tools, I’d ideally like to have it be that the world can keep moving regardless of what occurs.

My main design pattern idea for this is something I’m going to dub ‘Network-First State’. The idea of this pattern is, any change that needs to be made to state always goes direct to network instead of being written to the state. Then, when the host receives the written variables back from network serialization, those get written into state. That is, the behaviour loop for critical progression scripts goes like so:

+-------------------------+        +----------------------------------+
|                         |        |                                  |
|    Host code occurs     | -----> | Variables are written to network |
|             (host only) |        |                      (host only) |
+-------------------------+        +----------------------------------+
            ^                                       |
            |                                       V
+-------------------------+      +-------------------------------------+
|                         |      |                                     |
|    Client code occurs   | <--- | Variables are received from network |
|                         |      |                                     |
+-------------------------+      +-------------------------------------+

This way, the update code is only allowed to operate based on state that all other parties also receive, or get reasonably updated on. That way, all users have enough information to act as the host, but they are ‘sleeper’ hosts, in that they don’t actually run any of the host code or write to network.

If the host drops out, Udon will automatically move the ownership onto someone else, and they will thus automatically begin running the host code to keep the session up.

This isn’t really any different from how a typical P2P session might work for a conventional game, but it requires some special attention in Udon due to how data has to be synced.

In a way, this design reminds me almost of a torrenting seed pool. Udon automatically moves the owner to the oldest player when a player drops out, so having the host “seed” the data to everyone else results in the lowest loss occurring when the host drops out. That next player can then take on the mantle of “seeding” the data to everyone else.

In any case, this ended up being a lot longer of a write-up than I expected. I wanted to document some of my thoughts and experiments with Udon, and I figured I should probably use this website given I went to the trouble of setting it up.

I’m not very used to writing up long form stuff in this way, so I apologize if this has been a bit of a stream of conscious text wall, but I hope there’s at least something to take away from it. At the very least, I can come back to it when I’ve forgotten everything in a few weeks.