window_shape cheat sheet

This is a "cheat sheet" for "window_shape" extension by YellowAfterlife.
The extension can be found on itch.io.

Click on sections to expand/collapse them.
Quick display controls: Categories · Sections · Everything · Toggle night mode

Concepts

This extension offers control over how the window looks in a number of ways:

Shapes: these use SetWindowRgn and GDI regions.
Chromakey: mark a single color to be transparent.
Per-pixel alpha: partial transparency, as if your window was a surface.
Alpha: a simple multiplier that works with any of above

On shapes/regions

GDI regions are vector-based (consisting of rectangles, polygons, and ellipses) rather than pixel-based, which means that you can't have semi-transparent pixels on the window edge, but you can do binary operations on shapes (like combining them together or applying a mask).

If you've ever worked with vector graphics editors, this should sound familiar.

Some functions consume the inputs, meaning that the underlying native object is destroyed and the reference is invalidated - you don't have to call window_shape_destroy on them and you cannot use them in further functions.

Changing a window's shape forces a redraw, so you'll usually want to apply a shape once per frame at most.

Setting up

The underlying API is primarily intended for use with borderless windows.
While you can use it with regular windows, doing so degrades the window to some kind of Win2K-era appearance... you have to see it yourself, really.
You can enable borderless window mode in
- GMS1: Global Game Settings - Windows - Graphics
- GMS2: Game Options - Windows - Graphics
Import the extension to your project:
- GMS1: Right-click "Extensions" in resource tree, pick "Import", pick the GMEZ file.
- GMS2.2: Drag-and-drop the YYMP file onto the workspace area of the IDE.
- GMS2.3: menu:Tools - Import Local Asset Package
If you are using GM2024.14 or newer, you'll want to tick the "Use legacy DXGI_SWAP_EFFECT_DISCARD" checkbox in Game Options ➜ Windows ➜ Graphics.
Call window_shape_init() somewhere on game start / before using other functions.
Create a shape
Apply it to the window

window_shape_init()

Initializes the extension.

Should be ran before doing anything else.

Creating shapes

Coordinates are in pixels relative to window's top-left corner.

window_shape_create_empty()➜

Creates an empty shape (formally a zero-size rectangle).

Basic shapes

window_shape_create_rectangle(x1, y1, x2, y2)➜

Creates a rectangle shape.

window_shape_create_round_rectangle(x1, y1, x2, y2, w, h)➜

Creates a rounded rectangle shape (w, h being corner radii).

window_shape_create_ellipse(x1, y1, x2, y2)➜

Creates an elliptical shape.

window_shape_create_circle(x, y, radius)➜

Creates a circular shape, by which I mean: this is just a shorthand for the ellipse shape.

Polygons

window_shape_create_polygon_from_array(array, mode, ?count)➜

Takes an array of x,y coordinate pairs and creates a polygon shape out of it.

mode can be:

window_shape_polygon_mode_winding
The polygon is filled normally.
window_shape_polygon_mode_alternate
XOR-like logic is applied to polygon when it intersects itself.

count is the number of points in the array.
If not specified, it is set to (length of the array)/2.

For a practical example, if you do

var arr = [];
var angle = 30;
var mode = window_shape_polygon_mode_alternate;
repeat (5) {
    array_push(arr,
        200 + lengthdir_x(200, angle),
        200 + lengthdir_y(200, angle),
    );
    angle += 360/5*2;
}
window_shape_set(window_shape_create_polygon_from_array(arr, mode));

with "alternate" mode, the star would have a hole in the middle, while with "winding" mode it won't.

window_shape_create_polygon_from_buffer(buffer, mode, count = -1)➜

Same as above, but accepts a buffer with series of buffer_s32 x,y coordinate pairs.

If count is -1, it is set to buffer_tell(buf) div 8.

Buffer can be safely destroyed afterwards.

window_shape_create_polygon_from_path(path, mode)➜

Creates a shape from a GameMaker path.

Supports smooth paths somewhat accurately.

""Pixel""" shapes

By which I mean, a pile of rectangles (one per row of connected pixels).

These are kind of slow and I strongly encourage you to use polygons instead.

window_shape_create_rectangles_from_rgba(buffer, tolerance, width, height)➜

Creates a shape from pixel data in a buffer - any pixel with opacity above tolerance will be filled.

width and height specify dimensions, in pixels.

window_shape_create_rectangles_from_surface(surface, tolerance)➜

A convenience wrapper for above, creates a shape from pixels in a surface.

window_shape_create_rectangles_from_sprite(surface, tolerance)➜

A convenience wrapper for above, creates a shape from pixels in a sprite.

Shape manipulations

window_shape_duplicate(shape)➜

Creates an independent copy of the given shape.

window_shape_destroy(shape)➜

Destroys the given shape.

Transformations:

window_shape_shift(shape, xoffset, yoffset)➜

Moves the shape around by a specified offset.

window_shape_transform(shape, m11, m12, m21, m22, dx, dy)➜

Transforms a shape through a 2d matrix, but... the documentation for underlying function says that this converts the shape to rectangles first (??), so the results vary.

The following would give the window an appearance of a slightly skewed circle, for example

var _circle = window_shape_create_circle(100, 100, 100);
var _transf = window_shape_transform(_circle, 1, -0.2, -0.2, 1, 50, 50);
window_shape_set(_transf);
window_shape_destroy(_circle);

Binary operations:

window_shape_combine(shape1, shape2, op)➜

Combines the two shapes into a new one and returns it.

The inputs are consumed in the process.

op can be one of the following:

window_shape_operation_and
The result is the overlap between the shapes.
window_shape_operation_copy
The result is the first shape.
window_shape_operation_diff
Subtracts shape2 from shape1.
window_shape_operation_or
The result is a union between two shapes.
window_shape_operation_xor
The result is an exclusion between two shapes (overlaps are empty).

See MSDN for an illustrated example of what each mode does.

The following would give the window an appearance of a donut (a circle with a hole in the middle):

window_shape_set(window_shape_combine(
    window_shape_create_circle(100, 100, 100),
    window_shape_create_circle(100, 100, 40),
    window_shape_operation_diff
));

window_shape_combine_nc(shape1, shape2, op)➜

Like window_shape_combine, but does not consume the shapes.

window_shape_concat(shape1, shape2, op)➜

Like window_shape_combine, but the result is kept in shape1 instead of creating a new one.

The second shape is consumed in the process.

Returns whether successful.

window_shape_concat_nc(shape1, shape2, op)➜

Like window_shape_concat, but does not consume anything.

Utilities:

window_shape_contains_point(shape, x, y)➜

Contains whether the point is within a shape.

For purposes of checking whether the mouse is over a custom-shaped window, you will probably want to use display_mouse_get_x() - window_get_x() rather than window_mouse_get_x() (which would only update while the mouse is over the window).

window_shape_contains_rectangle(shape, x1, y1, x2, y2)➜

Contains whether a rectangle is within a shape.

Setting and resetting

window_shape_set(shape)

Applies the given shape to the game window.

This function consumes the shape.

That isn't my idea - the WinAPI function has it that after the call the region is confiscated from you and the system will manage it (including destroying it once the window closes or a different region is applied).

window_shape_set_nc(shape)

Like above, but does not consume the shape (by giving WinAPI a copy rather than the original).

window_shape_reset()

Returns your window to its original, rectangular form.

Alpha

This is a simple multiplier that applies over any other window shape manipulations.

window_set_alpha(alpha)

Changes the game window's opacity.

alpha should be in 0 .. 1 range.

window_get_alpha()➜

Returns the game window's current opacity.

Chromakey

This can similarly be used with other shape modifiers.

window_set_chromakey(color)

Changes the game window's chromakey color.

Pixels that fully match this color will be see-through and click-through.

Pass -1 instead of the color to disable chromakey.

window_get_chromakey()➜

Returns the game window's chromakey color (-1 if disabled).

Per-pixel alpha

window_enable_per_pixel_alpha()

This function calls DwmEnableBlurBehindWindow for a zero-sized region.

Doing so also happens to enable partial transparency support for the contents of the window ("The alpha values in the window are honored").

For example, if you did

draw_clear_alpha(c_black, 0);

in a Draw event of the only instance in a room, your window would look like an empty frame.

For a proper setup, make sure that "Clear Display Buffer", "Enable Viewports" (with at least one view set up), and "Clear Viewport Background" are enabled for the room; see desktop_friend project for a slightly more sophisticated example.

Now let's talk about the conditions and caveats:

Premultiplied alpha

Windows expects the window pixels to have premultiplied alpha.

In short, instead of just color, the pixels should be colored with

merge_color(c_black, color, alpha)

For sprites, all recent GameMaker versions have a "Premultiply alpha" checkbox in the Texture Settings;

For text and primitives, you'll need to use the above formula;

For mixing multiple semi-transparent images together, see the classic GameMaker surface problems and solutions.

If you don't use pre-multiplied alpha, your window will still work, but the semi-transparent bright portions will appear lighter than they should be.

Hit testing

According to Windows, just because a pixel is now almost or completely transparent, that doesn't mean that you can't click it!

You can use the extension's other functions (be it shapes or chromakey) to narrow down the window's shape, or use the window_set_clickthrough function from my free Window Commands extension to make the window clickable/un-clickable based on where the cursor is.

One-way ticket

The function has been around since Windows 7, yet it doesn't seem like you can go back to an opaque window once you have called it - even if you pass dwFlags of 0.

You can "top-up" the opacity of the window contents by drawing a black rectangle with bm_add blend mode or any rectangle with gpu_set_colourwriteenable (or draw_set_colour_write_enable for GMS1) of (0, 0, 0, 1).