Would it be better to set the camera direction to the ray direction and position then do the typical matrix operations on geometry and then check to see if the 2D geometry covers the center of the screen than the typical intersection tests for the GPU? I guess you could do a whole triangle at a time too, just gotta check however many pixels the triangle covers from around the center of the screen.

OpenGLs mouse picking worked this way: Set up projection matrix to mouse pointer position, render a tiny framebuffer and check which triangle is visible.
But ofc. this is not efficient at all, because it loops over all triangles, culls them, rasterizes them.

Not sure what you mean with ‘typical intersection tests’ on GPU, but if it's using a acceleration structure like BVH to speed things up, that's much faster.
With AMD adopting to HW RT soon, i think we don't have anymore reason to implement this ourselves. You would just use DXR, Vulkan, or the interfaces of U-engines around that to get fast RT.