Edit: Basically, I'm looking to get the roll angle from a forward (a vector tangent to the surface of a sphere) and up vector. The yaw and pitch are gotten using the up vector.

I have some game pieces on a sphere, and I move them along pseudorandomly-generated geodesics (great circles). The only problem is that the pieces do not face forward, in the direction of its tangent vector. I tried many different things, but I cannot figure it out. Note that the fractal is aligned along the z axis.

The whole code is at: https://github.com/sjhalayka/opengl4_stlview_shadow_map

The pertinent code is:

	void set_transform(void)
	{
		vec3 temp_dir = normalize(geodesic_dir);

		float yaw = 0.0f;

		if (fabsf(temp_dir.x) < 0.00001 && fabsf(temp_dir.z) < 0.00001)
			yaw = 0.0f;
		else
			yaw = atan2f(temp_dir.x, temp_dir.z);

		float pitch = -atan2f(temp_dir.y, sqrt(temp_dir.x * temp_dir.x + temp_dir.z * temp_dir.z));

		static const mat4 identity_mat = mat4(1.0f);

		const mat4 rot0_mat = rotate(identity_mat, yaw, vec3(0.0, 1.0, 0.0));
		const mat4 rot1_mat = rotate(identity_mat, pitch, vec3(1.0, 0.0, 0.0));
		
		const mat4 translate_mat = translate(identity_mat, temp_dir * displacement);

		const vec3 tangent_transformed = normalize((rot0_mat * rot1_mat) * vec4(normalize(geodesic_tangent), 0.0f));

		const mat4 rot2_mat = rotate(identity_mat, acos(dot(normalize(geodesic_tangent), tangent_transformed)), vec3(0.0, 0.0, 1.0));

		model_mat = translate_mat  * rot0_mat * rot1_mat * rot2_mat;
	}

So, you have a given random point on the sphere surface, and a given tangent direction where the object should look.
And you want a orientation matrix from those two?

If i get this right, it is easier to construct this matrix directly instead using rotations:

Matrix4x3 mat;
mat[0] = normalize(geodesic_dir); // tangent
mat[1] = normalize(temp_dir); // make up vector (which must be orthogonal to geodesic_dir, otherwise we would need to orthonormalize...)
mat[2] = cross(mat[0], mat[1]);

mat[3] = temp_dir * displacement;

Hey, thanks a lot for your help!

It seems to be working when I do this:

vec3 ndir = normalize(geodesic_dir);
vec3 ntan = normalize(geodesic_tangent);
vec3 nleft = cross(ntan, ndir);

mat4 mat;
mat[0] = normalize(vec4(nleft, 0.0f));
mat[1] = normalize(vec4(ntan, 0.0f));
mat[2] = normalize(vec4(ndir, 0.0f));
mat[3] = vec4(ndir * displacement, 1.0f);
		
model_mat = mat; 

What was obvious to me before you answered was that the solution would be simple and elegant.

What's not obvious to me is why it works. LOL. Anyone care to decipher the magic code?

OK, so m[0] is the 1st column, and it's a rotation along the x axis? Likewise for the 2nd and 3rd columns (rotations along the y and z axes, respectively)? The 4th column is a translation column. Altogether, it's effectively the product of 3 rotations and a translation?

Have you ever felt like a caveman who has been shown fire by ancient astronauts? LOL

OK, so m[0] is the 1st column, and it's a rotation along the x axis? Likewise for the 2nd and 3rd columns (rotations along the y and z axes, respectively)? The 4th column is a translation column. Altogether, it's effectively the product of 3 rotations and a translation?

Not quite. If you look at how you render those red/green/blue axis to visualize transforms, this tells the upper left 3x3 block of a transform matrix are just those 3 axis as visualized:

void VisualizeTransform (Matrix4x4 m)
{
RenderLine(m[3], m[3] + m[0], RED); // x axis 
RenderLine(m[3], m[3] + m[1], GREEN); // y
RenderLine(m[3], m[3] + m[2], BLUE); // z
}  

I know it's obvious - but think how this simple example demystifies all there is to know about how matrices represent orientation, and we could write our matrix class like this, and OpenGL would eat a pointer to that without issues:

struct Matrix4x4
{
vec4 xAxis;
vec4 yAxis;
vec4 zAxis;
vec4 position;
}

So with OpenGL conventions the first 3 rows map to x,y,z basis vectors directly, and nothing stops us from setting them up directly as well.
Usually we make sure all those vectors are orthogonal to each other and have unit length.
In case of dong something like constructing a look at matrix, we can set one axis to the direction (object - target), make the second close to the given up vector but ensure it's orthogonal to the first, and set the 3rd as the cross product of both.
In this example (very similar to yours) we do not need to do any rotations at all - we just construct the orientation geometrically by thinking of 3 basis vectors individually instead of an somewhat abstract matrix.

One common source of confusion here comes from the term ‘rotation’:

so m[0] is the 1st column, and it's a rotation along the x axis?

This is wrong, because to express a 3D ‘rotation’ we need all 3 axis together to make sense out of it.
But it would be correct to say: The first row is the x axis of the 'orientation’.

So this is not to nit pick. I think it's often helpful to imagine ‘orientation’ but not ‘rotation’. It's the same difference than between ‘point’ and ‘vector’. And using this, we can fix this too:

Altogether, it's effectively the product of 3 rotations and a translation?

Which is also wrong - there is no product of rotations at all, and we do not have 3 of them in our code. (You seem to confuse this with rotations using Euler Angles. But that's really the worst way to think or work with rotations, usually.)
No rotation. All we did was ‘constructing the orientation geometrically’. You can imagine a rotation from identity matrix to our result, but there is no need for that - it's easier to just skip over this and focus only on orientation. Without any movememt or change.

What's not obvious to me is why it works. LOL. Anyone care to decipher the magic code?

If my somewhat psychological attempt worked and my assumptions are right, you should be able to answer this yourself now. But let me know otherwise… : )

Right on. So they specify an orthonormal basis? Yes I am still fixated on Euler angles. As you can tell, I’m still a beginner at linear algebra. I have two linear algebra textbooks, but I have trouble reading them. Thanks for everything JoeJ, I really appreciate it.

taby said:
So they specify an orthonormal basis?

Yes.

Though, if we want this:

The basis is no longer orthogonal, but the basis vectors are still there and we can work with them.

As you can tell, I’m still a beginner at linear algebra.

Hehe, you're not alone : ) One obstacle for me is indeed that we can understand almost all math related to transformations purely geometrically, which i do. If you know what dot and cross product is doing, even matrix multiplication can be fully understood without even thinking about rows, columns, or a system of multiple equations.

OK, I didn't realize how easy linear algebra was, until you showed me the way. If I understand correctly, matrix multiplication is like a bunch of dot products? Thanks so much! Here is an image with the basis for each model in black:

I'm sorry to bother you again, but do you mean columns when it comes to assigning vec4s to the matrix elements m[0], … , m[3]?

I am looking at my old matrix code, and the fourth column has the eye translation in my homebrew version of gluLookAt. So OpenGL is column-major, right?