Realtime Displacement Maps using CanvasTextures in React Three Fiber

What is a displacement map?

This is a very valid question if you have never come across them before. It is a grey scale image, where white === higher and black === lowest. The material class in THREE has a displacementMap prop which we can use to apply the displacementMap (can be referred to a height map also - a little confusing).

Usually you would use a powerful engine like blender or houdini to generate these grey scale images pre run time. But I wondered if you can update this performantly in some way in real time and you can!

N.B. The dashed line drawing on the canvas context is not mine im just using it to show an example of drawing a line / curve over time.

Workflow - canvas

So how is this possible without image editing software to create grey scale images on the fly?

Drawing a line on a canvas context is pretty straight forward (with a bit of googling) and so is drawing this line over time. This jsfiddle is an example of this. Any you can modify the code to display on the map prop of the material to prove its working and showing.

But how does this help?

Well now we have an animation, all we have to do is somehow get a texture we can use in three/R3F everytime we increment the drawing in the context (using - JavaScripts setInterval). And in the color format expected - grey scale.

We can set the colors in these lines:

Copy 
1const context = canvasHolder.getContext("2d");
2if (context) {
3    context.rect(0, 0, canvasHolder.width, canvasHolder.height);
4    context.fillStyle = "black";
5    context.filter = "blur(9px)";
6    context.fill();
7}
8
9// And ....
10
11context.beginPath();
12context.moveTo(100, 20);
13context.lineTo(200, 160);
14context.quadraticCurveTo(230, 200, 250, 120);
15context.bezierCurveTo(290, -40, 300, 200, 400, 150);
16context.lineTo(500, 90);
17context.lineWidth = 5;
18context.strokeStyle = "white";

The fillStyle, strokeStyle and blur control how our grey scale im age looks.

Canvas Textures in Three

Here is the page describing canvas textures in the three docs.

It accepts a canvas as a parameter and gives us a texture just how we can use any other texture in the R3F components / uniforms. And guess what we have a canvas element in the DOM outside of the R3F canvas element..

We can select this element and use this to draw to it (positioning this additional canvas offscreen, literally position fixed and right 100%).

And its that simple!

One small thing, I have to define the maps declaratively in the material jsx component like so:

Copy 
1<meshStandardMaterial displacementScale={1.1}>
2    <canvasTexture attach="map" needsUpdate />
3    <canvasTexture attach="displacementMap" needsUpdate />
4</meshStandardMaterial>

I think this is just like in the case where we setup buffer attributes for points bufferGeometry, we have to instantiate the textures. Not 100% sure on this, played around for a while to figure it out.

Update the Maps and Displacement Maps

So we essentially progress along a line and draw segments of it ; much like you animate a dashed svg line or used to (don’t know if theres another way of animating svgs these days). We do this over time using setInterval and clearing when we get to 100% progress

Copy 
1const drawLine = () => {
2    //this clears itself once the line is drawn
3    lineInterval = setInterval(updateLine, 1);
4};
5
6function updateLine() {
7    //define the line
8    defineLine();
9
10    if (progress < length) {
11        progress += speed;
12        moveDash(progress, dir);
13
14        if (canvas.current) {
15            if (displacementMeshRef.current) {
16                const texture = new THREE.CanvasTexture(canvas.current);
17
18                displacementMeshRef.current.material.map = texture;
19                displacementMeshRef.current.material.displacementMap = texture;
20            }
21        }
22    } else {
23        clearInterval(lineInterval);
24    }
25}

Pretty simple setInterval setup and clearing the same interval.

And we pass our canvas to canvasTexture every time the interval callback is called. Then use this to set the displacement and the map properties on the planes displacementMeshRef set below this.

Copy 
1const texture = new THREE.CanvasTexture(canvas.current);
2
3displacementMeshRef.current.material.map = texture;
4displacementMeshRef.current.material.displacementMap = texture;

The scale is controlled by displacementScale prop on the material for the plane.

And that pretty much it!

This has some pretty powerful applications, like real time terrain tools and real time manipulation of vertices. How would you paint onto mesh and affect its colors or vertices?

Heres a former article which you might find useful.

Until next time 🙂