I have a function which works correctly with asCALL_CDECL_OBJFIRST. self is the correct pointer, and passed via rcx as specified in the x64 calling convention:
staticvoid ScriptTextureTest(OverlayTexture* self)
Now I also have this function:
static glm::vec2 ScriptTextureGetSize(OverlayTexture* self)
This one is broken - "self" is pointing to the rdx value passed in rather than the rcx value. MSVC seems to treat this function as though rcx is making room for 8 bytes of return value (2 floats in glm::vec2), and so self is assigned rdx, and thus the wrong pointer.
Microsoft's calling convention documentation mentions …