Quantcast
Channel: GameDev.net
Viewing all articles
Browse latest Browse all 17063

x64 calling convention doesn't make room for return value in rcx

$
0
0

I have a function which works correctly with asCALL_CDECL_OBJFIRST. self is the correct pointer, and passed via rcx as specified in the x64 calling convention:

staticvoid ScriptTextureTest(OverlayTexture* self)

Now I also have this function:

static glm::vec2 ScriptTextureGetSize(OverlayTexture* self)

This one is broken - "self" is pointing to the rdx value passed in rather than the rcx value. MSVC seems to treat this function as though rcx is making room for 8 bytes of return value (2 floats in glm::vec2), and so self is assigned rdx, and thus the wrong pointer.

Microsoft's calling convention documentation mentions …


Viewing all articles
Browse latest Browse all 17063

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>