The overall problem:

For advanced lighting in somewhat traditional renderers (without ray bouncing) we still need to sample various volumetric / area lights. The prevailing method, MRP (Most Representative Point) volume sampling, suffers from glaring artifact edge cases that over the years nobody fixed.

I’ll focus here specifically on the case of tube lights / line lights, because I recently needed them in a strict no-compromise scenario. But the same issues exist on e.g. Quad Lights (especially if you expect to make them 3D volumetric).

(You can also see the results in my open source openxr framework.)

Here’s a TL;DR video:

No I mean it, all the fancy papers, engines, shadertoys, rockstar authors (who for the record I believe to be smarter than me), have the same artifacts. Not throwing anyone under the bus but go to literally any pretty shadertoy or engine that features line or tube lights, and cycle the specular component, point the light at these odd angles against a wall and a sphere, change your view angles, and you see these deal-breaking problems.

The solution:

My shader is here and the function is MRPointOnTubeLight with plenty of comments, but let’s discuss high level. (todo: make cleaner shader for the article 🙃).

We need to handle all these things and their problems:

1. Horizon intersection

This is well documented. Handle the case where the light intersects with a horizon or surface (see frostbite paper). Basically if one (but not both) ends of the light are behind the surface normal, we find the closest point on the line that is the same side of the surface normal. The most optimized way is to subtract from that point a dot-projection-based distance.

This works but keep in mind any caps or distortions you might add to the line / shape.

2. 1st MRP for specular component

This is sold as the only thing you need for speculars. High level idea is straight-forward: spec = reflection. So find the closest point on the line in its reflection on the fragment.

See Frostbyte’s paper (search for MRP), or Alex Tardif, or Redorav, or Unreal / Brian Karis, or Unity. And Linear-Light Shading with Linearly Transformed Cosines (Eric Heitz and Stephen Hill) which has excellent visualizations. Some of these sources, e.g. Karis, Frostbyte, Unity, mention they know of (some of) the artifacts and simply accept them.

Problem

The concept of this math doesn’t work when the direction of the fragment to l0 and/or l1 is approaching dead-on the same as the direction of the reflection vector. And the tspec term causes a hard cutoff at the ends of the line/segment. This is not an edge case as you might imagine. As you saw in the videos, the distribution is wide and you notice the problem even at >45 degree angles:

1. There’s a “cylindrical edge” hard falloff artifact in the reflection on the side of the line light that’s furthest away from the fragment point (which should be the blurriest side not the sharpest).
2. And you get an “empty center hole” that you need to cap (duh there’s no line to pick a point on anymore).

3. 2nd MRP for diffuse component

Same idea, but the point is defined by the middle: the bisector of (frag->l0, frag->l1) touching l0->l1. Nicely visualized in Redorav’s article, or Linear-Light Shading with Linearly Transformed Cosines (Eric Heitz and Stephen Hill).

Problem

Practically the same issue. Can’t work when the direction of the fragment to l0 and/or l1 approaches the direction of the reflection vector. So you get big holes along the distribution of fragments. And because the diffuse is, well, diffuse, large, you also see it break towards the curve-horizon when you hold your light “parallel” to a sphere.

4. The caps

So if you want a volumetric light, you need to add (volume and) caps and this would hopefully fix the 2nd “empty center hole” problem. Right?

Check page 18 “Adding the End Caps” in Linear-Light Shading with Linearly Transformed Cosines (Eric Heitz and Stephen Hill) but then also page 21 “The caps approximation is not always worth it and can result in visually disturbing artifacts.”

I see those artifacts and raise you another artifact, which happens even without the caps! In the video on the left half:

The horizon skew problem: when the reflected line is ~parallel to the surface (e.g. ground), and the light is big / long / far away, the reflection’s end of the line that’s closest to you, appears to have a downward sharp slope at the very end. And the end furthest from you gets an upward slope distortion.

Yes everyone seems to also have this problem. Unless they cap the cylinder or pinch it somehow.

So we have:

1. a “cylindrical edge” hard falloff artifact
2. an “empty center hole” that you need to cap
3. a horizon sphere falloff problem
4. a capping problem
5. a horizon edge skew that happens without the cap

5. A 3rd MRP and patches

My magical but annoying fixes:

• Specular term: We shorten the ends of the tube distribution exponentially in a tight and roughness-aware way (glorified pows and lerps towards the ends of the line). After some effort it nicely rounds the ends, but shortens the caboose/furthest end of the tube reflection a tiny amount. Hey it’s not more inaccurate than the aberrating original.

• Everything: We pick a 3rd MRP that’s a simple point / sphere light position equivalent. And we try to include it where the other MRPs fail e.g. at the closest point on the tube to the surface. But, now we’re blending between one MRP and another MRP, based on angles, whereas IRL you get “infinite MRPs”. So because we got MRP1 <-> MRP3 for diffuse, MRP2 <-> MRP3 for specular, it won’t blend perfectly out of the box. But it’s close! I only needed to go into lerp hell to make sure I cat-heard all the math consistently across different smoothness / roughness and size and distance.

• For better caps / ends: You can fade the intensity of the tube light towards its ends in some exponential curve way so it muffles the ends only, then substitute with the point or sphere light. We even have enough math already to tell which end of the light (reflection) is closest to us. For tentacles I just faded to 0 by some pleasing curve, without caps.

Drawbacks:

• “It’s not accurate”. Oh, you mean the non-energy-conserving, artifact-ridden dealbreaker industry-standard MRP solution, is now not accurate despite now looking more or less flawless? (well I agree, it’s not accurate)
• It’s a little heavier, innit? Hell the cool kids just use the one specular MRP and hand pick values & poses, and pretend they got good lights. So fake it as much as you can I guess.

6. Add radius, distort the n dot l, and aPrime roughness/radius.

Well documented already, check shader and references. Keep in mind any caps or distortions you might add to the line / shape.

Then plug the light direction, distance, aPrime, nDotL, into your brdf function, and attenuate.

Conclusion:

I can even warp / animate / wiggle it a little bit before it starts breaking at the seams (ie the tentacles), so I’ve managed to make a reasonably™️ flawless™️ and accurate™️ volume light, across all roughnesses. Especially for XR / VR you can’t get away with less. And as a nice bonus, the whole thing plugs into whatever BRDF functions you already have / want to use.

So the (patched) MRP technique, has its uses. But by this point I’d rather do more proper raytracing into some sparse scene light data. Because it’s too high maintenance and it got heavier compared to what you thought you’d be doing when you heard about the easy MRP geometric shortcuts. Not a fan of solutions that rely on black holes of tweaking or too much smoke.

PS: The scene runs on a RTX 4070 mobile at 90FPS (the refresh rate of the headset) with 8 (animated) tube lights and 2 directional lights, and a lot of transparent objects overdraw. It also runs on a RTX 2060 mobile with a bit of lag spikes if you have too much overdraw right on your face.

TL;DR:

Performant open-source OpenXR, Vulkan, ECS, C++. A boilerplate -> framework -> mini “game engine” to quickly make an actual playable modern royalty-free XR game.

Demystifies ECS / Memory Management, Single Pass Rendering, XR Input, and XR gamedev fundamentals, on top of @janhsimon’s excellent timesaving khronos setup openxr-vulkan-example.

[Github: https://github.com/tdbe/openxr-vulkan-gamedev-framework]

(There’s also a youtube hq 1600x1600/1440p 60fps version.)

Abstract:

*Trey Parker voice* Vulkan has a rich body of work, and many strengths as a people; but lack hoo-man compatibility. I’ve managed to translate their stack, for hoo-mans, whose lifetimes may otherwise be too short to first decipher the khronos lunar manuals for hope of achieving even the most basic useful contact.

It didn’t help that they don’t want to touch Single-Pass rendering (the performant & industry-standard linchpin of rendering).

In any case, thanks to open-source you can now build something pretty good the right way, without worrying about mighty morphing license agreements or wetting the beaks of people with golden parachutes.

Builds:

• See Build-ProjectSetup.Readme.md or just be lazy and run the windows build in ./out/ (or the github Release).

• “Recommended Specs and Minimum Requirements”: The scene runs on a RTX 4070 mobile at 90FPS (the refresh rate of the headset) with 8 (animated) tube lights and 2 directional lights, and a lot of transparent objects overdraw. It also runs on a RTX 2060 mobile with a bit of lag spikes if you have too much overdraw right on your face.

Controls:

• Controls.md

My feature stack so far:

(sections ordered from high-level to low-level)

XR Locomotion

• Rotating and (accelerated) Panning of the scene by grabbing with both hands, retreating into a non-euclideanly warped pocket dimension (pushing the world away from you non-linearly) and seeing a “tunnelvision” portal-style chaperone. Highest effectiveness and lowest sickness (carefully tweaked and tested across dozens of different people).
• Uses state machines for movement and for visuals. Supports animated teleportation with targets.

Base XR gameplay mechanics

• Mechanics system based on a list of GameBehaviours set up as FSMs.
• Each behaviour is Created (with its own required references), Updated (with frame & input data etc), and Destroyed.
• Mechanics for locomotion, hands, XR Input testing, world objects.
• Any GameComponent has one or more GameEntity parents and manual or automatic cleanup (and preventing dangling components when all owners are freed). But there’s no parenting between different game entities / objects, so manipulate groups of matrixes yourself. TODO: add a parenting system that processes the chain of Transform matrixes.

Physics

• There’s support for running jobs on Bounds components (generated at model load time), with proper functions for AABB intersection or enclosure tests, plane tests, rectangular selection (even at non-Axis-Aligned angles) / frustum casting, raycasting.
• There’s a concept of ground in the locomotion system.
• But TODO: no actual Physics library added.

Animation

• *crickets* TODO: just add it via full/extended gltf support.
• TODO: find something open-source for: IK, gpu-skinning, and LoDs.

Audio

• *crickets* TODO: add Audio component, and threaded spatial sound library with audio queueing.

GUI

• *crickets* TODO: vector or sdf text, and just textures. Then use previously made ui code.
• TODO: main menu, in-game hands inventory
• (I’m certainly not implementing a scene-graph management system (game editor))

Jobs / Threading

• The objects and memory is set up in a spanned ECS manner but TODO: no job system / threading example (there’s only sequentially updated GameBehaviours / state machines on the main thread).
• TODO: add a simple chunking concept, run jobs in parallel on chunks.

GameData

• Everything is set in generic memory-span pools, by type. You set up a game world with maximum allocated memory for each pool, then during gameplay you can request to use a free object, or mark a used one as free and reusable. There’s no need for defragmenting, or swap-and-pop (would be slower in average-case) (“ted talk” in GameDataPool.h).
• Enities and components are based on GameDataId (serving as a weak reference): [globalUIDSeed][index][version] and a top-level [typeIndex] for convenience.
• Everything is easy to request and keep track of through various means, even by name in hash maps for light scripting purposes.
• Cleanup is either manual (and cache coherent) or automated via (cache-missing) awareness of component dependencies.
• GameEntity and GameEntityObject
• GameComponent: Material, Model, Transform, Bounds, Light.
• Properties: isVisible, isEnabled, name, some events etc.
• PlayerObjects {GameEntityObjects, PlayerActiveStates}.
• Materials {Shader, Descriptor-set UniformData, instancing, optional/shared Pipeline (for e.g blend ops)}

Rendering

• Implemented the most high quality e.g. Disney BRDF lighting equations for diffuse, specular and MRP based (Most Representative Point) shape lights.

• TODO: does not include clearcoat.
• TODO: does not include subsurface scattering,
• TODO: does not include shadows,
• TODO: no per-pixel transparent object sorting,
• ^, ^^, ^^^: but, I’ll someday add in raytracing into some form of nanite clumps or other semi-volumetric discrete mesh data, instead of going through the legacy shading/sorting timesinks again.
• Per-material, per-model, per-pipeline properties. Easily create a material e.g. transparent, doublesided; add new shaders with all the static and dynamic uniform data you need, instance geometry etc.
• Render pipeline knows if you modified any default properties and creates pipelines from unique materials. Tries its best to batch per unique material and per model to minimise GPU-CPU communication, and has instancing, but it’s not Indirect Rendering.
• Expanded, added to, and explained Khronos’ & JanhSimon’s Headset, Context, Renderer/Pipeline etc, and the easily misleading & hard to customize khronos vulkan <-> openxr implementation. Especially regarding multipass vs singlepass & multiview, and what it takes if you want to use your own renderer or a diffrent API like webgpu. (look for "// [tdbe]" )

Input class and InputData.

• A ‘proper’ universal xr input class, supporting (probably) all controllers/headsets, with customizable binding paths and action sets.
• Nicely accessible data through InputData and InputHaptics, including matrixes and other tracked XR positional data.
• Poses for controllers and for head.
• Actions (buttons, sticks, triggers, pressure, proximity etc).
• User presence / headset activity state.
• Haptic feedback output.
• Exposes action state data (e.g. lastChangeTime, isActive, changedSinceLastSync)

Utils

• Utils and math for XR, input, general gamedev.
• Debugging.
• TODO: Bring in gizmos, e.g. debug lines, wireframes for lights etc.

A typical run log:

• Game world load, setup, updates & render loops, unload and exit. TypicalRunLogSample.md

Additional Attributions

⠀

⠀

Below is Janhsimon’s original readme:

Overview

This project demonstrates how you can write your own VR application using OpenXR 1.1 and Vulkan 1.3. These are its main features:

• Basic rendering of example scene to the headset and into a resizeable mirror view on your desktop monitor.
• Focus on easy to read and understand C++ without smart pointers, inheritance, templates, etc.
• Usage of the Vulkan multiview extension for extra performance.
• Warning-free code base spread over a small handful of classes.
• No OpenXR or Vulkan validation errors or warnings.
• CMake project setup for easy building.

Integrating both OpenXR and Vulkan yourself can be a daunting and painfully time-consuming task. Both APIs are very verbose and require the correct handling of countless minute details. This is why there are two main use cases where this project comes in handy:

1. Fork the repository and use it as a starting point to save yourself weeks of tedious integration work before you get to the juicy bits of VR development.
2. Reference the code while writing your own implementation from scratch, to help you out if you are stuck with a problem, or simply for inspiration.

Running the OpenXR Vulkan Example

1. Download the latest release or build the project yourself with the steps below.
2. Make sure your headset is connected to your computer.
3. Run the program!

Building the OpenXR Vulkan Example

1. Install the Vulkan SDK version 1.3 or newer.
2. Install CMake version 3.1 or newer.
3. Clone the repository and generate build files.
4. Build!

The repository includes binaries for all dependencies except the Vulkan SDK on Windows. These can be found in the external folder. You will have to build these dependencies yourself on other platforms. Use the address and version tag or commit hash in version.txt to ensure compatibility. Please don’t hesitate to open a pull request if you have built dependencies for previously unsupported platforms.

Attributions

Open Source ECS Project

Documented here, and source code at: https://github.com/tdbe/Tdbe-2023-URP-DOTS-ECS-Graphics-Physics.

v 2022.2.6f1

.’s 1.0 sandbox

• everything is unmanaged, bursted, and (multi)threaded by default (except the legacy input)
• player, game system, states, random/spawners, variable rates, threads, aspects, dynamic buffers & nativearray components, collisions, dynamic bounds, warping, pickups with visuals, rocks, ufos+ai, shooting and health / dying.

• 5-8 ms on main thread, and 140-190FPS, with 500k-1m triangles

• Diagram of the ECS layout: https://miro.com/app/board/uXjVMWg58OI=/?share_link_id=616428552594

Project

Fairly wide scope of ECS DOD / DOT usage, generic and specific setups, for a full game core loop. There are still a few gameplay details done in a hurry, marked with “// TODO:” or “// NOTE”.

The project is set up to be visible via the Hierarchy what is going on and roughly in what order, and using prefabs and components with mono authorings, and inspector notes. Most things & rationales are also described in comments in code.

Assets[tdbe]\Scenes\GameScene_01 <– scene to play

Play:

• Control ship with the chosen input data keys in the corresponding Player prefab. By default:
  • Player_1: arrow keys to move (physics like a hovercraft), right Ctrl to shoot, right Shift to teleport. Touch the pickups to equip them.
  • Player_2: WASD to move, Space to shoot, leftShift to teleport.
• To easier test, you have 1000 health. You get damaged by 1, every physics tick, by every damage component that is touching you. Anything that dies disappears, no animations, but there is health GUI.

Some points of interest:

• everything is physics based.
• I made what I think is a cool multithreaded RandomnessComponent using nativeArray, persistent state, and local plus per-game seeds.
• simple but reusable Random Spawner aspect, also reused in targeted spawning of child rocks and player teleportation.
• resizeable window / teleport bounds
• equipped pickups are visible on you and have modding behaviour to your health or to your shooting (e.g. go through objects).
• tweakable health and time to live on everything that moves including rocks.
• tweakable damage dealing from everything that moves.
• randomized PCG for variable rate update groups, randomized (and/or binary) sizes as well, for enemies and rocks.
• enemy AI follows closest player, even through portals (picks shortest path to nearest player, including portals).
• Quickly made a dumb but cleverly versatile offsetted outline shadergraph shader that I quickly built all my assets from “CSG style”.

Philosophy:

• performant (threaded, bursted, instanced, masked) by default, not “well this won’t hurt so much”.
• main system can update states of other systems, other systems control their own state and do their one job. (and there can be sub-branching).
• a system changes the component state a thing, and then another system takes over. E.g. no scripting of events chains on spawn or calling systems etc.
• reuse components, systems, threads, and aspects, unless doing so becomes confusing project-management wise or future-gamedev wise. #ProgrammerUX is real.
• at the same time don’t preemptively expose code that you don’t need anywhere else yet. E.g. you can even use “{ }” blocks locally, in some large main function (yeah I know usually one function does one thing). Don’t end up with confusing directionless fragments for someone else to hunt down and wonder when to use, etc.
• use state machines, state graphs; some approaches are described in code (e.g. in GameSystem). Before starting a big project, create a state / decision transition visualizer that your grandma would understand.
• break up large / often edited parts of components for iteration & cache coherency, have a look at the chunk buffers.
• track game’s memory limits. Pay attention to when anything (should be) increased / destroyed and queue them only in specialized systems at safe times.
• track all the other bandwidths / stress points :) (threads, hardware, non-ecs-ties); e.g. what happens if you wipe out all enemies on the screen at the same time?
• make philosophy clear at a glance: hierarchy object naming and structure, inspector notes, code descriptions of intention or the point etc.
• In ECS anything can be represented as just an efficient database query. So the difficulty, the limits & wisdom, are about how you store, define, equip, and see this query as a state or concept, in a production-friendly sane way.

Some annoying quirks I found:

• At this time cross-scene communication techniques in unity ECS are: *crickets* ..just use statics or somehtin..?
• Oh what’s that, you just wanted to quickly access some main Camera data, from your entity subscene? 🙃
• Yo what’s up with Variable Rate Update Groups insta-updating on rate change and not next tick!?
• Some things you wouldn’t expect, don’t get authored from mono. For example: isKinematic, isTrigger, physics layers.
• Rigidbody freeze position and rotation do NOT have a solution from Unity in ECS. Yeah there’s the external JAC shit but it’s not the same behaviour, it’s restricting and sometimes physics-unreliable AF joint authoring components.
• Yes you knew about the renderer and TransformSystemGroup when spawning, but did you know/remember the ECS Fixed Step Physics simulation will also process entity colliders at 0,0,0 if you don’t use the right command buffer stage.
• NonUniformScale/PostTransformScale component (to be replaced with PostTransformMatrix) is not disabled but actually absent by default, and can be requested / added.
• Getting collision hit points. I get it, but very cumbersome UX…

You probably know that you can assign and write to render textures and 3D textures and even custom data buffers like RW Structured Buffers from Compute Shaders. These can hold spatial information trees, distance fields, flow maps, points, meshes etc. (For more reading look up UAVs (Unordered Access Views), SRV (Shader Resource Views), and shader Resource Binding registers.)

But with shader model 5.0 and d3d11 you can now do more or less the same in regular vertex fragment shaders. This is great because it allows you to easily bake data onto mesh textures or atlasses while they’re being rendered to screen anyway.

It’s got such a cool range of possibilities, that you can even do dumb simple stuff like sample the shader’s color under your mouse cursor, from inside the shader, and write it from inside the shader to a struct that you can simultaneously read back in C#, with no need to iterate through pixels or have access to any textures or anything like that.

So I’m’a show you how to UAV in unity shaders.

Number 1: Constructing Render Targets

Your shader will have a RWTexture2D (or even 3D if you wanna get fancy and bake some point clouds):

1
2
3
4
5
6
7
8
9
10
CGINCLUDE
#ifdef _ALSO_USE_RENDER_TEXTURE
	#pragma target 5.0
	uniform RWTexture2D<half4> _MainTexInternal : register(u2);
	
	sampler2D sp_MainTexInternal_Sampler2D;
	float4 sp_MainTexInternal_Sampler2D_ST;
#endif
//... other stuff
ENDCG

The register(u2) represents which internal gpu registrar to bind the data structure to. You need to specify the same in C#, and keep in mind this is global on the GPU.

Now you can use this _MainTexInternal as if it was a 2D array in your shader. Which means it will take ints as coords like so _MainTexInternal[int2(10,12)] - which means it won’t be filtered / smooth. However, you can form C# assign this same RenderTexture as a regular Sampler2D texture in the material/shader, with material.SetTexture as you would with any other texture, and then you can read from it with regular UVs.

So now let’s create that render texture in C# and assign it to the material. Do this in a ConstructRenderTargets() and call it from something like Start().

1
2
3
4
5
6
7
8
9
10
11
12
if(m_MaterialData.kw_ALSO_USE_RENDER_TEXTURE)
{
	m_paintAccumulationRT = new RenderTexture(rtWH_x, rtWH_y, 0, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Linear);// Must be ARGB32 but will get automagically treated as float or float4 or int or half, from your shader code declaration.
	m_paintAccumulationRT.name = _MainTexInternal;
	m_paintAccumulationRT.enableRandomWrite = true;
	m_paintAccumulationRT.Create();
	
	m_MaterialData.material.SetTexture(m_MaterialData.sp_MainTexInternal, m_paintAccumulationRT);
	m_MaterialData.material.SetTexture(m_MaterialData.sp_MainTexInternal_Sampler2D, m_paintAccumulationRT);
	Graphics.ClearRandomWriteTargets();
	Graphics.SetRandomWriteTarget(2, m_paintAccumulationRT);//with `, true);` it doesn't take RTs
}

On that last line above, note the nuber 2. That’s the register index from the shader. So register(u2) corresponds to 2 here.

Number 2: Constructing Data Buffers

Let’s just create an array of some arbitrary MyStruct, that will exist in both the shader and in C#.

1
2
3
4
5
6
7
8
9
10
11
12
13
CGINCLUDE
#ifdef _ALSO_USE_RW_STRUCTURED_BUFFER
	#pragma target 5.0 // no need to re-declare this directive if you already did it 
	
	struct MyCustomData
	{
		half3 something;
		half3 somethingElse;
	}
	uniform RWStructuredBuffer<MyCustomData> _MyCustomBuffer : register(u1);
#endif
//... other stuff
ENDCG

So RWStructuredBuffer<MyCustomData> is our buffer. It has some limits of what can go inside, and it’s not 100% the C standard. But it’s still really useful and can hold tons of entries or just a few (as much as a texture, or as much as memory allows).

Now let’s construct the Compute Buffer in C#. Do this in a ConstructDataBuffers() and call it from somehting like Start().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
//Needs to be defined the same as in the shader.
public struct MyCustomData
{
	Vector3 something;
	Vector3 somethingElse;
}
MyCustomData[] m_MyCustomDataArr;

void ConstructDataBuffers()
{
	if(m_MaterialData.kw_ALSO_USE_RW_STRUCTURED_BUFFER)
	{
		int memalloc = 24;
		m_MyCustomComputeBuffer = new ComputeBuffer(bufferLength, memalloc);//stride == sizeof(MyCustomDataStruct)
		Graphics.SetRandomWriteTarget(1, m_MyCustomComputeBuffer, true);
		m_MaterialData.material.SetBuffer(m_MaterialData.sp_MyCustomBuffer, m_MyCustomComputeBuffer);
		
		m_MyCustomDataArr = new MyCustomData[bufferLength];
	}
}

If y’all know how to iterate through memory, you know what that memalloc value is for. It’s the size of the struct in bytes. A float3 is 12 bytes, and the structure I created in the shader has 2 half3’s which equal to 1x float3 :) In the C# side, we don’t have half3s but we can define it as Vector3 which is resolved to float3 and is bound as a half3 on the GPU in our case. If you’re worried about conversion, there’s a Mathf.FloatToHalf() function.

Now to do stuff with the data of this struct from C#. Do this in Update() if you want/need, it’s fine.

1
2
3
4
5
6
7
8
9
10
11
12
13
void ProcessData(){
	if(m_MaterialData.kw_ALSO_USE_RW_STRUCTURED_BUFFER)
	{
		//here's how to read back the data from the shader
		m_MyCustomComputeBuffer.GetData(m_MyCustomDataArr);//obviously this way you will loose all the values you had in the array beforehand
		
		m_MyCustomDataArr[10].something = new Vector3(1,0,1);
		
		//now set it back to the GPU
		m_MyCustomComputeBuffer.SetData(m_MyCustomDataArr);
	}

}

Now to do stuff with this data buffer on the shader side:

1
2
// somewhere in vert or in frag or in a geometry function or wherever:
_MyCustomBuffer[my1DCoorinate].somethingElse = half3(0,1,0);

Done! Now go do cool stuff. And show me.

One of the things I did with this technique was a VR mesh painter where I have a custom SDF (signed distance field) volume to represent a 3D spray volume function intersecting with the world position on the fragment of the mesh I’m drawing on. You can also atlas your UVs so that you can have your RT as a global atlas and paint multiple objects to the same RT without overlaps.

You also need to realize that the objects are painted from the PoV of the camera, and so it might not hit fragments/pixels that are at grazing angles if you use say a VR controller, and you’re not aiming down the camera’s view direction. This results in sometimes grainy incomplete results depending on the mesh and angle. But you can fix that by doing the rendering with a different camera mounted to your hand and so you can render the painting passes of your obects only, with ColorMask 0, invisibly, to that camera. (or just using compute shaders isntead).

You can also do this whole thing but with Command Buffers and DrawMesh instead of Graphics.Set… I’ve done this a couple of times using Compute Shaders, but with the 5.0 vert frag shaders I had issues tricking unity’s API to work last I tried.

So perhaps another blog post should be about how to set up command buffers and compute shaders, and how to do something cool like turn a mesh into a pointcloud and do cool ungodly things to it :)

It’s called Voice of God, and the player shapes the ground. We won 3/5 awards in the ITU-Copenhagen site!

Among the prizes we got was a ham and NGJ tickets!

The theme was WAVES this year and we used your voice to make waves in the ground. Roughtly, your pitch (the frequency) is mapped to the screen from left to right (low pitch to high), and your loudness affects the amplitude.

A little more gameplay! It works surprisingly well! #GGJCPH pic.twitter.com/cb4TJSTAjC

— Bjarke A. Larsen (@BjarkeAL) 22 January 2017

You’re effectively controlling the rolling character via a winamp visualization. The cleaner your sound and the smoother your vocal range, the more effective you are.

We used unity 2D tools, the Animator, a rigidbody pool, and sound basetones and overtones merged into a world position function.

Voice of God itch.io page here. Github repo with build here or here. And the original GGJ submission’s page is here.

It was a lot of fun but punishing to our vocal chords after the 2 days of testing and development :)