Introducing: GMLive.gml!

An animation of live-coding a visual effect in a game "Voidigo" (mouseover/click to play GIF) From Voidigo | full sized video

Roughly this time last year, I released GMLive - a web-based program that allows to write and run small snippets of GameMaker code right in the browser, including live reloading.

Today I'm pleased to announce that situation further improved and I am bringing live-coding to GameMaker: Studio and GameMaker Studio 2 themselves.

Table of contents

1. What is this
2. Technical details
2.1. Compiler
2.2. Interpreter
2.3. Server
2.4. Other notes
3. Using GMLive!
4. Could YoYo Games do this themselves?
5. Releases

What is this

Game development often requires multiple iterations to get something right.

With things like user interface or game mechanics, it is extremely common to tweak code or parameters decades of times for them to be just right. So you change something, run the game, look at it, close the game, change it again, run the game...

Needless to say, that can be a bit of a bother, especially since compile times grow longer as projects grow bigger, and it can take time to get the game into a state where you can observe subject of testing if you did not spend time making some general-purpose debugging tools.

A solution to that is interactive programming - updating parts of a program while it is running.

Now, unless your programming language of choice was specifically designed with this functionality (or general dynamic evaluation) in mind, you cannot simply have live-coding with it - in case of GameMaker, recent versions (GameMaker: Studio, GameMaker Studio 2) no longer ship games with complete source code and compiler+interpreter packed in, and do not natively support "side-loading" additional game logic.

As result, the only way to make interactive programming a reality with GM was to implement a custom compiler+runtime for GML that would be overwhelmingly compatible with the original language. Certainly not an easy task, but still a one that I was able to pull off. So let's talk about that for a bit...

Technical

The whole thing can be separated into several distinct modules:

Compiler

A compiler must take source code, verify correctness of it, and produce data for interpreter to run.

This bit hadn't actually changed much, and you can see post about GMLive for technical details.

The main difference here is that instead of generating JavaScript you generate bytecode.

The interpreter is implemented as a stack-based VM (as opposed to register-based), which saves from some headache when implementing a bytecode compiler.

So, for example, if you had

if (some) {
	trace("hi!");
}
trace("done!");

it would be compiled to something like (shortened pseudocode):

push(self.some);
if (!pop()) goto(label_1);
	push("hi!");
	trace(pop());
label_1:
push("done!");
trace(pop());

If you were to add an else-branch,

if (some) {
	trace("yes");
} else {
	trace("no");
}
trace("done!");

the output would be

push(self.some);
if (!pop()) goto(label_1);
	push("yes");
	trace(pop());
	goto(label_2);
label_1:
	push("no");
	trace(pop());
label_2:
push("done!");
trace(pop());

For operators and alike,

trace(a + b * c);

the output is also linear,

push(self.b);
push(self.c);
push(pop() * pop());
push(self.a);
push(pop() + pop());
trace(pop());

Overall it's mostly about having solid logic in place for everything, but still a handful of work.

Interpreter

This is where things get interesting.

While GameMaker's extension system has seen improvements over past years, some things remain platform-specific or inaccessible entirely - such as access to raw variable data, instances, or calls to built-in functions or scripts from extensions.

As result, the most reliable method of implementing an interpreter that can access entirety of GML is to have it done in GML itself. Or, in my case, making Haxe generate GML code to save time.

If you have a plenty of experience with various programming languages, the act of writing an interpreter might look like an easy part - first you decide on a set of bytecode instructions to cover the syntax that you want to support (GMLive has 75 of these in total),

And then you write a function that has a semi-massive switch-block in it, doing more or less exactly what each instruction defines,

But wait. There's a small detail here: GML has no exception handling.

Which means: If your interpreted code throws up an error, the game does as well.

Would you want to re-launch your game just because you typo-ed a variable name? I can make a confident assumption that most people wouldn't. And thus your semi-massive switch-block becomes a semi-massive switch-block with a pile of error handling in it:

Error handling in GMLive.gml source code
If by this time you're wondering what the generated GML is like, please reconsider.

But wait, that's not all - you probably want to be able to call built-in functions as well. But you can't get a "pointer" to a built-in function, nor resolve them by name. Additionally, passing wrong argument types into functions is a no-no, same as storing result if there isn't one returned.

The solution to this pile of trouble is to have a separate little script for every function so that you can reference scripts instead. However, GMS has over 1600 functions, so writing that many by hand is generally out of question. Therefore another piece of software was developed, which takes a giant file with function definitions (some taken from fnames file, some hand-edited),

and generates an even larger GML extension with one script+declaration per function.

Overall, a task of moderate complexity but far greater time investment.

Server

One could assume that producing a pretty accurate GML runtime for GML itself would be all you need for a livecoding extension, but, alas - more technical tasks:

  • Some resource types (such as shaders) cannot be enumerated in-game at all.
  • Macros (in GMS2 only), enums, and globalvars can be defined wherever.
    (having GML scripts parse other GML scripts isn't a good option)
  • All game scripts have to be scanned to resolve the number of arguments for them.
    (to catch at least some argument count related runtime errors)
  • Scripts must be scanned before their first execution to ensure that the updated code is available by then.
  • A game cannot access it's project directory as such due to sandboxing rules.
  • Any approach taken must work on both Windows and Mac (for GMS2 support).

In light of these conditions, a decision was made to develop an external piece of software that would perform all these tasks and deliver the required information for the game.

The result was a Haxe-Neko VM application that communicates with the game over HTTP.

The server spots a compiler of it's own (since macro/globalvar/enum detection must be contextually aware), although one can be called a "half-compiler", as it is smart enough to tell apart comments, strings, arguments (, ...) as it quickly looks over the entirety of project code, but makes no attempts of handling code structures that it doesn't need (which is most of them).

A particular advantage of having a separate server application is that you can test over network - for example, you can start up the server (and allow it in firewall), configure the game to connect to your local IP address, start the game on the device in same network, and have it update in real-time as you experiment with touch controls.

Sending a game build to a collaborator and having them give input while you tweak their game right as they play it is also an amazing thing.

Other notes

Originally, the extension was going to be GMS2-only. The reason for that was pretty simple - GMS1 doesn't let you re-save project while the game's running.

As I continued to show off demonstrations of GMLive, it became increasingly apparent that people do want such a thing for GMS1 as well, and it should be worthwhile to investigate the workarounds.

After a handful of testing it was discovered that not all is lost - in fact, GMS1 would specifically wait for the game's process to exit, and not lock the project files (which meant that other programs could still change them). The actual reasoning behind this is unknown, but I assume that it is simply to avoid dealing with more than one game trying to send debug information to IDE at once.

As result, several workarounds were implemented and documented, and all was well.

Using GMLive!

As for something pretty magical, an expected question is - "but how easy is it to set up and use?". The answer to that is - really easy. In fact, you can set everything up within 1 minute:

Setting up GMLive in GameMaker Studio 2 (mouseover/click to play GIF) Setting up GMLive in GMS2 | full sized video

So you import the package, import the assets (an object, a few included files, and the actual extension) from it, and add the object to your first room.

Prior to running the game, you run the server from included files and add live_call lines to events/scripts that you want to be reloaded on change.

And that's it - you can now run the game, change some code, save it, and that change will be reflected in-game as soon as GMS is done updating the file on disk.

GMS1 version takes a little longer to set up (mostly because of limitations of extension system in the version), but still easily doable.

Could YoYo Games do this themselves?

And now for the semi-obligatory section for posts that outline me doing what is perceived to be someone's else work.

Honestly - probably! YoYo Games is a [semi-]big company, which employs a number of skilled programmers, some of which are certainly more skilled than me.

However, as a big company, YYG also has obligations, such as fixing bugs or streamlining the processes to make the average developer's life easier, so I wouldn't be surprised if a feature like this has been on their backlog for years and never even made it onto roadmap.

Besides, it would probably require a major overhaul of their compiler code, which would be even more complex than mine.

Releases

The extension is available via itch.io and Marketplace:

itch.io GM:Marketplace

itch.io version ends up being ~25% cheaper due to rounding downwards after subtracting revenue cut difference (GM:Mc takes 30%, itch takes 10% or custom amount), so you can pick between paying a few dollars less and being able to quickly view/install the extension from the IDE.

While I expect some people to point out that the extension costs slightly more than they were expecting, the amount of work that went into it is tremendous, and it should be entirely possible to save up [what is less than a cost of an average boxed videogame] for something that can save you tens to hundreds of hours of work in long run.

Have fun!

Related posts:

8 thoughts on “Introducing: GMLive.gml!

  1. Hey yal, finally got some money to buy your extension. One point: performance.
    Tried it out with blank project where I added test object and enabled live call in it’s draw event. After rendering 20×20 array I have had 1 800 fps with live call disabled and 20 fps with live call on. That’s very unwelcome, any way you could improve this?

    • Depending on what exactly you were doing (should show code), assigning the array to a local variable (and generally using local variables where you can) can help noticeably, as that saves the interpreter from having to call variable_instance_get (which is still kind of slow) to fetch instance variables.

      GML currently does not support modifying variables by persistent reference/pointer, so there isn’t much that can be safely done about this without getting into some heavy grade code analysis (auto-caching instance variables into local variables), which would also have disastrous effects if any mistakes are made in algorithm.

  2. Vadim, your work in regards to GameMaker is absolutely remarkable. I cannot even find the words to describe how useful, how powerful this product is.This will revolutionize game development in GMS2, probably even the wholly sphere of game programming as a whole. Thank you!

  3. Pingback: A summary of my GameMaker assets

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.