My comments:
+ IFS is very useful, especially in stress-testing certain classes of graphics algorithms (the Brown Mesh Set includes more models than you'll ever need).

+ Don't support implicit casting from the math objects (vec*, quat, etc) to the appropriate pointer representation. Make the casting explicit. Between the variations in GL's functions and the various legal ways implicit conversions can occur, you run the risk of causing very serious bugs very easily. This is bad thing.

+ The D3DX mesh object sees two primary uses: usage as an actual vehicle for rendering geometry, and usage as an intermediate container for access geometry (e.g., loading the .X file then pulling the vertex/index buffers from it and discarding the mesh itself, to use the buffers manually). Keep that in mind; if you're concerned about making a mesh object because people might want to use their own custom ways of rendering geometry, a mesh object can at least be useful as a means of transforming on-disk geometry into in-memory vertex and index buffer representations.

+ Also note that .X is not a D3D-specific format at all (it is, in fact, extremely flexible and extensible). Although writing a proper loader is not exactly pleasant without the help of the D3DXFILE interfaces, so I'm not exactly suggesting you bother (COLLADA is probably better for that anyway). Just something to be aware of.