Interesting. Perhaps it is just RedHat that is doing something differently then.

Hopefully when I figure out the solution for the problem on Ubuntu it will work for Fedora as well.

I've identified what it is that causes the crash on Ubuntu.

The following line in the inline assembler code in X64_CallFunction is the cause:

" .cfi_def_cfa_register r15 \n"

Unfortunately I can't just remove this line, because that causes the crash to happen on RedHat instead.

I'm still trying to figure out how to determine when the line must be included and when not.

It turns out that it wasn't something specific to Ubuntu or RedHat after all.

Instead it was caused by compiler optimization. By default the library is built without any optimizations, but with my tests on RedHat I had turned on the optimizations some time back and forgotten about it. Turning off the optimization caused the problem to happen on RedHat too with the original code. Once I figured this out the solution was simple:

#ifdef __OPTIMIZE__
// Make sure the stack unwind logic knows we've backed up the stack pointer in register r15
// This should only be done if any optimization is done. If no optimization (-O0) is used,
// then the compiler already backups the rsp before entering the inline assembler code
" .cfi_def_cfa_register r15 \n"
#endif

Now everything works with or without optimizations turned on, and on both Ubuntu and RedHat. (Most likely it will work on Fedora too)

The fix is available in revision 2197.

Thanks for letting me access your Ubuntu machine. Without it I would most likely not have figured it out.