Preface: Sorry for the long and involved explanation. Basically, the problem I have is a Vertex shader behaving seemingly arbitrarily and texturing incorrectly, but I want to give lots of background to make solving the problem easier.

I am making a game with terrain in OpenGL and C. You can imagine the terrain first as a 2d grid of square tiles. Then, a) Each of the corners of these tiles have some height value, and b) Each tile has another vertex in the middle of it, the height of which is equal to the average of all the height values of the corners around it.

You can imagine a top-down view of one tile as a square with a cross through it, 4 triangles, so I am drawing each tile using a GL_TRIANGLE_FAN with six vertices: 1 for the center vertex, 1 for each of the 4 corners, and then a final one to loop back around to the first corner. My approach to making sure the texture vertices get chosen properly is not to use 2 extra floats in the vertex attribute and pass the whole UV to the shader for each vertex, but instead just add an extra byte to each attribute: A number from 0 to 4. 0 means that this is the center vertex, and 1-4 mean the tile corners.

Here is the code that assembles the Vertex Array. t_vec is an array of vectors of the full 3d positions of the vertices in the middle of the tiles, and c_vec is likewise for the corner vertices.

//Assembling the triangle fans
struct vbo_vert tri_fan[256][1536] = {0};
for (int i = 0; i < 255; i++) {
	for (int j = 0; j < 255; j++) {
		int j_6 = (j * 6);

		tri_fan[i][j_6 + 0]  =
			(struct vbo_vert){t_vec[i][j], 0};
		tri_fan[i][j_6 + 1]  =
			(struct vbo_vert){c_vec[i][j], 1};
		tri_fan[i][j_6 + 2]  =
			(struct vbo_vert){c_vec[i + 1][j], 2};
		tri_fan[i][j_6 + 3]  =
			(struct vbo_vert){c_vec[i + 1][j + 1], 3};
		tri_fan[i][j_6 + 4]  =
			(struct vbo_vert){c_vec[i][j + 1], 4};
		tri_fan[i][j_6 + 5]  =
			(struct vbo_vert){c_vec[i][j], 1};
	}
}

This gets handed to the GPU as-is, and I know that the Vertex Attrib Pointers are good because the terrain looks right geometrically. That's why I am pretty sure that the problem is in the Vertex Shader.

//VERTEX SHADER
#version 330 core
layout (location = 0) in vec3 a_pos;
layout (location = 1) in int a_texref;

out vec3 v_pos;
out vec3 v_col;

uniform mat4 model;
uniform mat4 view;
uniform mat4 proj;

void main()
{
	if (a_texref == 0) {
		v_col = vec3(1.0, 1.0, 1.0);
	} else if (a_texref == 1) {
		v_col = vec3(1.0, 0.0, 0.0);
	} else if (a_texref == 2) {
		v_col = vec3(1.0, 1.0, 0.0);
	} else if (a_texref == 3) {
		v_col = vec3(0.0, 1.0, 0.0);
	} else if (a_texref == 4) {
		v_col = vec3(0.0, 0.0, 1.0);
	}

	gl_Position = proj * view * model * vec4(a_pos, 1.0f);
	v_pos = a_pos;
}

//FRAGMENT SHADER
#version 330 core
out vec4 FragColour;

in vec3 v_pos;
in vec3 v_col;

void main()
{
	FragColour = vec4(v_col, 1);
}

You can see that I am not passing UVs to the fragment shader, but colours. This is for debugging purposes. The results are similar when I use UVs. The idea here is that the middle of the tile should get coloured white, then the top-left corner should be red, then the next three corners should be yellow, green, and blue in a clockwise(or maybe anticlockwise, depending on how I may have flipped this elsewhere) manner. Instead I see this:

It looks like the center vertex is being coloured correctly (white) but the outside vertices are all blue! But the weird thing that's freaking me out is that if I swap the bottom two rungs of the else if ladder in the vertex shader, not changing their conditions or code or anything but their position in the source, the tile edges all turn out green instead! It's as if the vertex shader is not bothering to actually check the conditions on any of the else ifs and is just choosing the one on the bottom, but except for the first vertex…

I can't explain this, and I'm wondering if someone can. I figure there's some quirk of the vertex attribute or shaders involving triangle fans or ints that I don't understand, or a typo or something else that I can't think of. I also have a github repo with all the code at this https://github.com/majorarchitect/game​ but I'm not sure if it will be needed.

When setting up your attrib pointers, are you using glVertexAttribPointer or glVertexAttribIPointer for the a_texref attrib?

Oh, I forgot about this thread!

21st Century Moose said:

When setting up your attrib pointers, are you using glVertexAttribPointer or glVertexAttribIPointer for the a_texref attrib?

It turns out that I should have used done this in my C code, as you said. I asked this somewhere else and someone gave me this important nugget of info.