Introducing: GMLive.js

For a little while now, I was working on a new thing - a program that would allow to test GameMaker code right in your browser. It is now complete, published, and is pretty cool.

You can either check it out right now or continue reading for development details and a bunch of demonstrations of it in action.

Technical, in short

So, in short, this works as following:

  • I wrote a GML->JavaScript compiler that functions closely enough to that of GM itself.
  • I found a way to resolve names of built-in functions in the compiled game and used this to make dynamically generated JS code call the built-in functions directly.
  • I wrote several Ace editor plugins to provide syntax highlighting and auto-completion on par to what GameMaker itself does.

Technical: Transpiling GML

When I mention "wrote a GML->JS compiler", that, of course, is a bit of simplification.

While GML is a relatively non-complex language (on par with JS), writing a complete parser and transpiler for it, let alone one that behaves closely enough to the official one, is a challenge.

Making it work right in browser - even more so.

But, not to worry - as you may already know, by this time I wrote quite a number of parsers, lexers, and compilers for all sorts of formats and languages, meaning that I generally know what I'm doing.

As far as actual compiler goes, it is written in Haxe, which remains one of the best options for this kind of thing. Code is converted into "tokens"; tokens are converted into an abstract syntax tree, syntax tere is checked and modified a little to ensure that all is well and to carry out required structure modifications. Finally, structure is converted to actual JavaScript code, and inserted into a separate small page running a special blank-but-not-quite game.

GML being akin to JavaScript, many structures translate pretty well. Some don't.

For example, GML spots a with-statement, which on a glance may look similar to what Object Pascal and JS have, but is in fact a loop - passing an object type instead of a instance reference will have it pick through all of them. Therefore,

with (obj_some) show_debug_message(id);

must compile to something like

var with_arr = find_instances(1/* obj_some */);
for (var with_key in with_arr) {
    if (!with_arr.hasOwnProperty(with_key)) continue;
    var with_inst = with_arr[with_key];
    // (several check-skips for instance being active)
    show_debug_message(with_inst._id);
}

Other distinct thing is GM's historic handling of arrays. See, in GML you are not required to initialize something as an array, you can do

var arr; arr[0] = 1;

And that will make a 1-element array. But not just that, you could also initialize arr to a non-array value, and that would still work. What is actually going is that GameMaker compiler finds out the first-of-a-batch array modification, and inserts a check before it, like so:

var arr;
if (!(arr instanceof Array)) arr = [];
arr[0] = 1;

Automatically locating the spots for inserting this takes a particularly curious algorithm that looks outwards and backwards of an expression while looking into branches along the way.

Overall, while reasonably challenging, I was able to get compilation to work very closely to original, meaning that any average script pasted into GMLive runs about the same as it would in GMS' HTML5 target.

Technical: Interfacing with GM:S HTML5 games

This is where the second act of curious things starts: by default, GameMaker minifies all function and variable names (for filesize concerns) in HTML5 builds, meaning that show_message will become something like _ZN1.

Obviously, setting original+renamed function name equivalents would have been very time consuming (there's over 1000 functions total!) and not future-proof (since adding new scripts/variables causes the rest to change names based on new sorting results).

But, not to forget - this is JavaScript. JavaScript' API has all sorts of features, some so specific that you may not even realize what they can be used for. For instance, you can get any JS function' code as a piece of text. Of course, with all variables and functions renamed, it may not make any immediate sense, but that is fine - it describes exactly what the browser will do upon calling it.

The other bit is what functions GameMaker renames and does not rename. There are three general groups of identifiers that GM will not rename on compilation:

  • Reserved JavaScript function, variable, and keyword names: Because not every value is produced by the game itself and you do not want any syntax errors.
  • Extension functions and variables: GM does a reasonable thing of not messing with extension code - extensions are output as-is, and calls to their functions are kept with original names.
  • User-defined scripts starting with gmcallback_. These were originally added for use with clickable_ functions (for producing button-like page elements that could call the game code), but can be used as-is as well (with a few gotchas).

As the result, if you make a script of kind

/// gmcallback_test
ext_func("func1", func1(1));
ext_func("func2", func2(1, 2));

It will be compiled to something like

function gml_Script_gmcallback_test() {
ext_func("func1", AQ1(1));
ext_func("func2", DF3(1, 2));
}

Which, combined with ability to get function code as a string, allows to very literally detect patterns in resulting JavaScript code to resolve the shortened names of functions.

So the seemingly-blank game spots a small extension which does it's share of trickery to map out every function of interest and store them in regularly-named global variables, which can then be referenced by generated code. This way the JavaScript generator only needs minimal knowledge of game state to output valid code, and the generated code is even pretty readable.

Inserted JavaScript then firstly calls a function that cleans up the results of previous execution, initializes what is needed, and starts the new cycle, calling JS scripts where needed.

All in all this is both easier and weirder than one could expect.

Technical: Editing GML

The next part is on making an appropriate editor.

While GML is not a complex programming language, syntax highlighting and auto-completion play a noticeable role when working with it.

For some of the previous projects I've used CodeMirror. CodeMirror is pretty good and easy to use, but is single-threaded, meaning that it does not cope too nicely with oversized files.

So, for this project, I've decided to use Ace. Ace is specifically designed for large tasks (makes use of Web Workers API for multi-threading) and is used in Cloud 9 IDE.

That said, Ace is very nice. While documentation does not cover a few things, source code is nicely documented, and I was able to find answers to all my subjects of curiosity via debugging or looking at the existing code (mostly of components in Ace itself).

Syntax highlighting was pretty easy to do - Ace's implementation is based on using regular expression groups to control tokenizer state, thus was a matter of using an existing set of highlight rules as reference to finetune GML-specific detection rules.

Auto-completion was a little trickier but still fine. To display argument list descriptions, these are stored separately and "unpacked" during initialization.

Status bar is a slightly altered version of built-in status bar plugin, which for mysterious reasons displays columns and rows starting from 0 by default. Function argument list is perhaps the largest highlight here, working by looking through tokens backwards from cursor position to discover whether the cursor is inside a function call.

User-defined scripts are highlighted, including showing arguments for them if they have documentation-comments at the start (/// script(arg1, arg2)).

Enums are currently not highlighted, as I am still figuring out the most appropriate method of having the application listen for structure changes (enums declarations being added/removed).

Overall, again, everything comes pretty closely to how GameMaker itself works.

Demonstrations

But - enough with technical talk. Let's move onto the interesting bits. Click on any of the images to load up the according demo and click "Run" to run it.

Let's start with basics - displaying a piece of text on screen:

trace function is much like the regular show_debug_message, but accepts a variable number of arguments. Additionally, both functions now displays a scrolling log of messages over the game view for convenience.

If you'd like to mess with drawing functions, you can - just #define a script called "draw", and it'll be called on every frame (note: rendering loop is disabled unless there's a "draw" script):

If you'd like interactivity, you can do that too, same as you would in regular GM:

(one thing to keep in mind is that you'll need to click on the game area before it can receive keyboard inputs)

Next up is 3D. It appears to be a little known fact that GameMaker allows to use majority of d3d_ functions on HTML5 (making use of WebGL). Similarly, the procedure is equivalent to what you would do in GameMaker: Studio itself:

While there are some quirks with 3d mode in HTML5 still, there's enough to be able to experiment. There's also support for mouselock functions from my extension if you'd like to try making something with first-person view or "mouselook" in general.

Finally, instances. You currently cannot declare new object types (an API for that was deprecated), but you can create instances as per usual, and do what you want with them:

As with other data structures, instances are cleaned up on program restart/shutdown, so you don't need to do any additional actions to quickly iterate on code.

Future considerations

Depending on tool's reception, popularity, and donations, there are various possible ways of further improving upon it:

  • Enum highlighting: As I've previously mentioned, it would be nice to provide syntax highlighting and auto-completion for enum fields aside of compile-time error checking.
  • Further editor polish: There are various small details that can be finetuned, such as auto-completion list popping up after 1 character entered (while in GM it pops up after 2) or display style for auto-completion items.
  • Asset UI: It would be more convenient if it was possible to add assets via some sort of user interface. This isn't a particularly critical issue though, since I was able to patch in support for Data URI scheme (e.g. base64) into most loading-related functions.
  • Object definitions: As previously mentioned, it would have been better if it was possible to define new object types. A good approach for dealing with this is not yet found, however.
  • Server-side JavaScript target: My work on GML->JS compiler opens up a theoretical possibility of creating a more complex compiler based on it, which would output code that could be ran on Node.js servers. Unfortunately, this is something that requires substantial time investment (thus funding), and is a bit unlikely to happen too soon.

In conclusion

This was not a small nor an easy project, but in the end it worked out, and works well.

There are still some things that can be done, but it perfectly usable.

You can support the development by donating via itch.io if you'd like.

Open GMLive Donate (via itch.io)

Related posts:

27 thoughts on “Introducing: GMLive.js

  1. Can you please make an android app out of it or make a mobile-friendly version? The text editor seems to be working in a very weird way. You can delete symbols one at the time, you can’t copy/paste/select all and the most annoying thing is that every new symbol replaces the previous one 50% of the time for some reason. Works fine on PC, though.

    • This seems to be a limitation of the code editor component I’m using (Ace). I don’t think wrapping this in a mobile app will fix anything because mobile browsers just do a plenty of things differently

      If you have free time, you can try doing something with the editor source code (shared between GMEdit and GMLive.js) to make it work better on mobile.

    • It cannot, as GM does not implement file dialog functions on HTML5, and a web page cannot read arbitrary files from a computer without explicit permit (or everyone would have been in a big trouble)

    • A couple helper functions got lost in translation while I was updating the entire thing to share UI with GMEdit so that won’t work for a bit. Equivalent is making a surface, filling it with color, and creating a sprite from it.

  2. Does GMLive.js work with vertex buffers? I’ve tried to create them in order to test something for my project I am working on, but your system shows errors.

  3. Pingback: Introducing: GMLive.gml!

  4. Yal, will you release source code for this? I’d like to study this amazing work of yours and possibly help with further development of this. Always when I come to your site I just wonder how you can be this way outsmarting everyone around.

    • GMLive itself likely won’t be seeing a source code release in near future, as documenting and cleaning it up [to a state where a non-expert user could work with it] could take a very long time. Would probably also have to do the same for dependencies, as it relies on a custom Haxe->JS compiler (for non-immutable algebraic data types), which relies on a custom library for Haxe AST manipulation, which relies on a pile of macros for generating optimized AST traversal functions…

      Needless to say, while I’m intrigued about the things that could be made with it given “a couple” months of full-time work, my own schedule does not have a magical ~3-month shaped hole in it, so there are no dates I can promise on that front.

      I did clean up my extensions for Ace editor though – see ace-bind (GM-like toolbar & auto-completion), ace-gml (GML syntax rules), gml-api (list of functions/variables/constants for both). These do have some differences to “vanilla” GML now (because I’ve added lightweight object support, template strings, named script arguments, GMS2-like macros, etc.) but it’s just some 1000 lines of code total so it shouldn’t be too bad.

      Have been also considering to write a post on writing flexible parsers+AST manipulators in Haxe (which would let above-intermediate users to try doing whatever they want instead of trying to figure out how my code works), but haven’t decided on structure or length of that yet. There’s kind of a lot of things to do at the moment.

  5. do you have the source for this available? would be great to extend to allow multiple instances, etc.

    • You can create instances – there are 10 predefined blank objects that you can mess with – example.

      I have plans to allow to set object’ event scripts and inheritance, but there are no GML functions for such things, so it would require reverse-engineering the generated code and writing an extension to dynamically locate and patch the internal functions. The existing code of similar purposes can be seen here. Needless to say, these things get weird quickly.

  6. Pingback: Introducing: Nuclear Throne Together v9877

    • Curiously, SilentWorks was experimenting with a browser-based version of Tululoo shortly before it was abandoned, but the volume of work turned out to be completely insane – not only would one have to rewrite the entire IDE (decades of thousands of lines of code), but also rewrite or find implementations for a plenty of components that are common on desktop but simply don’t exist on HTML5. And, if you were to do something with it now, also spend months to update it to current HTML5 standard. In other words, maybe you should learn Phaser + Phaser Editor or something…

  7. Pingback: GameMaker: get/set variable value by name

  8. Very cool Yal. Might be very nice to help people answer questions on the gmc too. I’m not sure how I can use it yet but it’s interesting to follow.

    • That’s a really good idea! If I ever need to help someone in terms of code, I know exactly what I’ll be doing now.
      And by the way Yal, this is absolutely amazing! I’ve always wondered if it was possible to get any kind of real-time editing done with GML, and well here it is, in all its glory. Great work.

  9. This is super cool!

    Is the ‘obj_blank’ and ‘spr_test’ the only default assets usable now, or how do you insert and or existing call assets?

    Super fun way to code in a browser.

    Why did you make this, or what is the goal? Just curious to know, how this relates to HTML5 and its application. But hey its almost now already a browser game.
    At a minimum, it will be a great way to demonstrate code examples live online!

    If assets are currently not possible to be easily added, we can just use some programmer art of course.

    I love this, since I know GML but not that much other browser languages and this will take my GML knowledge to more potential places. So consider making the canvas and code interchangeable, or like a card flip option also. So you can hide the code and only display the canvas area or frame…then it will have more application.

    I could not test the 3d demos in chrome :(. The voxels sounds great.

    • There are spr_blank (has no subimages), spr_white16 (a 16×16 white square), obj_blank (an empty object), obj_1..obj_9 (some extra empty objects if you want to use them for collisions), and fnt_test (used by default).

      You can add new sprites\backgrounds\fonts via the according built-in functions (sprite_add, background_add, …). Data URI (“data:image/png;base64,…”) format is supported.

      To test 3D you’ll need to switch mode (in toolbar) to GL – otherwise it uses a configured build that only has 2d functions defined.

      It is largely intended for testing code or prototyping small ideas.

      I’ll be looking into adding additional layout options in future. Currently it is intended to be open in a more or less maximized window with code editor occupying roughly half of it (as shown in demos).

    • I’ve now fixed the “messed up code” problem – turns out that some browsers decide to truncate big chunks of text in pop-up messages, replacing the middle with “…”. Made it show a custom small pop-up dialog instead.

      And that is cool! Even generates separate chunks.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.