Monogame RenderQueue designs

00. 32bit key

MSB                                                                   LSB
 00           06               14  15                 26               32
  ┣━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━╋━━━╋━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━┫
  ┃pass      64┃layer        256┃ 0 ┃effect         512┃texture      256┃  BATCHED MATERIAL
  ┣━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━╋━━━╋━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━┫
  ┃pass      64┃layer        256┃ 1 ┃data                       ~2 bytes┃  DYNAMIC MATERIAL
  ┣━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━╋━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
 00           06               14  15                                  32
MSB                                                                   LSB

     64  passes
    256  layers
    510  effects (0=null, 511=variable)
    254  textures (0=null, 255=variable)
17 bits  layer data


Designed for the Monogame framework, this sorts materials:
  1) by pass (background, foreground, camera, ui)
  2) by layer (level tiles, obscurable, overhang)
  3) any material renderable with a SpriteBatch
  3A)  by effect (since the batch flushes per effect change)
  3B)  by texture for a given effect (to minimize texture changes)
  4) any mayerial that does not use the SpriteBatch

Keys are calculated once per (pass, layer, material) tuple and should be
cached to minimize computation.  Batched materials with variable effects
or textures should set IsVariableEffect and/or IsVariableTexture to prevent
thrashing the sprite batch with extra flushes.

This grouping aims to pack renderable materials into the smallest number of
SpriteBatch flushes, while still allowing a material to specify a variable
texture and/or effect.

Each material is passed a RenderContext object and an opaque context object.

The RenderContext provides the following:
  (readonly) final RenderTarget
  (readonly) current pass Camera
  (readonly) current pass RenderTarget
  (readonly) current pass BlendState
  (readonly) current pass Layer (int)
  (mutable) current pass Effect
  (fn) PauseBatch
  (fn) ResumeBatch

The opaque object is usually specific to the Material subclass, although this
will depend on whatever is feeding the render queue.  For a simple
ECS -> Render Queue setup, a Rendering System could simply feed
(material, entity) tuples and let the material look up necessary components
(transform, auras, etc) to render the material.

01. 64bit key

MSB                                                                            LSB
 00           12   14                                                           64
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
  ┃pass        ┃ 00 ┃instruction                                                 ┃  PRE PASS
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
  ┃pass        ┃ 01 ┃layer       ┃ 00 ┃instruction                               ┃    PRE LAYER
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━╋━━━━╋━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
  ┃pass        ┃ 01 ┃layer       ┃ 01 ┃ 00 ┃instruction                          ┃      PRE BATCH
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━╋━━━━╋━━━━╋━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┫
  ┃pass        ┃ 01 ┃layer       ┃ 01 ┃ 01 ┃effect      ┃texture     ┃instruction┃        MATERIAL
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━╋━━━━╋━━━━╋━━━━━━━━━━━━┻━━━━━━━━━━━━┻━━━━━━━━━━━┫
  ┃pass        ┃ 01 ┃layer       ┃ 01 ┃ 10 ┃instruction                          ┃      POST BATCH
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━╋━━━━╋━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
  ┃pass        ┃ 01 ┃layer       ┃ 10 ┃instruction                               ┃    POST LAYER
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━┻━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
  ┃pass        ┃ 10 ┃instruction                                                 ┃  POST PASS
  ┣━━━━━━━━━━━━╋━━━━╋━━━━━━━━━━━━┳━━━━┳━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┫
 00           12   14           26   28   30           42           54          64
MSB                                                                            LSB

   4096  passes
   4096  layers
   4094  defined textures (0=null, 4095=variable)
   4094  defined effects  (0=null, 4095=variable)
50 bits  pass instruction
36 bits  layer instruction
34 bits  batch instruction
10 bits  material instruction

Designed for the Monogame framework, this is similar to the 32 bit
design with the addition of explicit pre- and post- steps for each pass
and layer.

In addition to expanding the limits of passes (64 -> 4096) et al. this
removes the concept of "batched" and "unbatched" since any non-batched
material may choose to render before or after the batch.  In the 32bit
design, a simplifying assumption was made that non-sprite batched
materials are always rendered later.

The pre- and post- instructions were envisioned for mutating the rendering
state machine, including render target creation and targeting commands,
changing effect parameters, adjusting compositing instructions, creating
and moving cameras, etc.

This can be made arbitrarily complex by allowing the pre- and post-
instructions to modify the current instruction pointer, either to skip or
repeat previous rendering steps.  For simplicity of reasoning and
implementation, modification of the render queue is not recommended.


Finally, note that all of this can be achieved in the 32bit version by
(ab)using the combined (pass, layer) index tuple to provide a maximum of 16k
"virtual passes" and simply packing a 17bit function pointer into the "data"
portion of a dynamic material.

02. sample class design

The following pseudo classes demonstrate one way to
unpack and process a set of (key, material, ctx) tuples
from a generic ECS.


// ========================================
// Rendering
// ========================================

RenderContext
    FinalRenderTarget
    PassRenderTarget
    Layer
    Effect
    Camera
    BlendState

    PauseSpriteBatch()
    ResumeSpriteBatch()

    // SpriteBatch Draw overloads, omitting depth arg
    // since that is defined as (Layer / MaxLayer)
    Draw(Texture2D texture, ..)

Material
    Texture2D
    Effect
    bool IsVariableTexture
    bool IsVariableEffect

    abstract Render(RenderContext ctx, object o);

Material<T> : Material
    void Render(RenderContext ctx, object o) => Render(ctx, (T)o);
    abstract void Render(ctx, T o);

RenderQueue
    void Begin()
    void Render(int key, Material m, object o);
    void End()

    int CalculateKey(int pass, int layer, Material m);


// ========================================
// ECS
// ========================================

RenderingSystem
    RenderQueue rq;
    EntitySet entities;

    RenderingSystem(RenderQueue renderQueue) => rq = renderQueue;
    RenderComponent Create() => new RenderComponent(rq);

    void Render() {
        rq.Begin();
        foreach(var e in entities) {
            var c = e.Get<RenderComponent>();
            for (var (k, m) in c.MaterialPairs) {
                rq.Render(k, m, e);
            }
        }
        rq.End();
    }

RenderComponent
    RenderQueue rq;
    List<(int k, Material m)> MaterialPairs;
    
    RenderComponent(RenderQueue renderQueue) => rq = renderQueue;

    void AddMaterial(int pass, int layer, Material m) {
        MaterialPairs.Add((rq.CalculateKey(pass, layer, m)), m);
    }

BasicEntityMaterial: Material<Entity>
    void Render(ctx, Entity e) {
        var t = e.Get<Transform>();
        ctx.Draw(Texture, t.Position, ..);
    }