[arm64] /usr/bin/luajit: bad light userdata pointer

[cdluminate]

https://buildd.debian.org/status/fetch.php?pkg=lua-torch-torch7&arch=arm64&ver=0~20170511-gae1a805-1&stamp=1495277863&raw=0

set -e; cd deb.test; \
	LD_LIBRARY_PATH=/<<PKGBUILDDIR>>/deb.test \
	/usr/bin/luajit -e "require 'torch'; torch.test();"
/usr/bin/luajit: bad light userdata pointer
stack traceback:
	[C]: in function 'typename'
	./torch.lua:114: in function 'setdefaulttensortype'
	./torch.lua:153: in main chunk
	[C]: in function 'require'
	(command line):1: in main chunk
	[C]: at 0xaaaac75ac538
debian/rules:64: recipe for target 'override_dh_auto_test' failed
make[1]: *** [override_dh_auto_test] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
debian/rules:30: recipe for target 'build-arch' failed

[cdluminate]

that is to say, this doesn't work for arm64:

torch7 : init.lua

153 torch.setdefaulttensortype('torch.DoubleTensor')

114       torch.Storage = torch.getconstructortable(torch.typename(torch.Tensor(1):storage()))

[cdluminate]

https://buildd.debian.org/status/fetch.php?pkg=lua-torch-torch7&arch=arm64&ver=0%7E20170720-gaed3171-1&stamp=1501263690&raw=0

The arm64 build is still failing. I'm going to disable the arm64 (aarch64) build.

[Handsome8]

I encountered the same problem, have you solved it?

[GitBubble]

openResty also met this issue:
openresty/lua-nginx-module#1152
do we really need some nasty hacks to make it work on ARM64 ?

Mike Pall's suggestion:

LuaJIT on 64 bit systems provides a limited range of 47 bits for the legacy lightuserdata data type. This is only relevant on x64 systems which use the negative part of the virtual address space in user mode, e.g. Solaris/x64, and on ARM64 systems configured with a 48 bit or 52 bit VA. Avoid using lightuserdata to hold pointers that may point outside of that range, e.g. variables on the stack. In general, avoid this data type for new code and replace it with (much more performant) FFI bindings. FFI cdata pointers can address the full 64 bit range.

[GitBubble]

since luajit wont fix this issue, application itself need to be adapted to avoid using of lua_pushlightuserdata,

in order to make Torch7 works on ARM64, following files need to be rewritten.

update_on_definition in ctype.c (extra\luaffifb) : lua_pushlightuserdata(L, &to_define_key);
update_on_definition in ctype.c (extra\luaffifb) : lua_pushlightuserdata(L, &to_define_key);
set_defined in ctype.c (extra\luaffifb) : lua_pushlightuserdata(L, &to_define_key);
set_defined in ctype.c (extra\luaffifb) : lua_pushlightuserdata(L, &to_define_key);
push_upval in ffi.c (extra\luaffifb) : lua_pushlightuserdata(L, key);
set_upval in ffi.c (extra\luaffifb) : lua_pushlightuserdata(L, key);
ffi_metatype in ffi.c (extra\luaffifb) : lua_pushlightuserdata(L, &user_mt_key);
push_user_mt in ffi.c (extra\luaffifb) : lua_pushlightuserdata(L, &user_mt_key);
luaopen_libcutorch in init.c (extra\cutorch) : lua_pushlightuserdata(L, state);
hookf in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, (void )&KEY_HOOK);
hookf in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, L);
gethooktable in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, (void )&KEY_HOOK);
gethooktable in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, (void )&KEY_HOOK);
db_sethook in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, L1);
db_gethook in ldblib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, L1);
hookf in lib_debug.c (exe\luajit-rocks\luajit-2.1\src) : lua_pushlightuserdata(L, KEY_HOOK);
LJLIB_CF in lib_debug.c (exe\luajit-rocks\luajit-2.1\src) : lua_pushlightuserdata(L, KEY_HOOK);
LJLIB_CF in lib_debug.c (exe\luajit-rocks\luajit-2.1\src) : lua_pushlightuserdata(L, KEY_HOOK);
lj_cf_package_require in lib_package.c (exe\luajit-rocks\luajit-2.1\src) : lua_pushlightuserdata(L, sentinel);
lj_api.c (exe\luajit-rocks\luajit-2.1\src) line 697 : LUA_API void lua_pushlightuserdata(lua_State L, void p)
ll_require in loadlib.c (exe\luajit-rocks\lua-5.1\src) : lua_pushlightuserdata(L, sentinel);
lua.h (exe\luajit-rocks\lua-5.1\src) line 170 : LUA_API void (lua_pushlightuserdata) (lua_State L, void p);
lua.h (exe\luajit-rocks\luajit-2.1\src) line 171 : LUA_API void (lua_pushlightuserdata) (lua_State L, void p);
lua.h (exe\luajit-rocks\luarocks\win32\lua5.1\include) line 170 : LUA_API void (lua_pushlightuserdata) (lua_State L, void p);
lua.h (exe\luarocks\win32\lua5.1\include) line 170 : LUA_API void (lua_pushlightuserdata) (lua_State L, void p);
lua.h (install\include) line 171 : LUA_API void (lua_pushlightuserdata) (lua_State L, void p);
luaT_iscdata in luaT.c (pkg\torch\lib\luaT) : lua_pushlightuserdata(L, CDATA_MT_KEY);
luaT_iscdata in luaT.c (pkg\torch\lib\luaT) : lua_pushlightuserdata(L, CDATA_MT_KEY);
json_process_value in lua_cjson.c (extra\lua-cjson) : lua_pushlightuserdata(l, NULL);
lua_cjson_new in lua_cjson.c (extra\lua-cjson) : lua_pushlightuserdata(l, NULL);
parse_record in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_name_key);
parse_record in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_name_key);
append_type_name in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_name_key);
append_type_name in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_front_name_key);
append_type_name in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_back_name_key);
find_canonical_usr in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_name_key);
find_canonical_usr in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_front_name_key);
find_canonical_usr in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_back_name_key);
find_canonical_usr in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, &g_name_key);
calculate_constant1 in parser.c (extra\luaffifb) : lua_pushlightuserdata(L, P);
add_tmpname in paths.c (pkg\paths) : lua_pushlightuserdata(L, (void)tmpnames_key);
add_tmpname in paths.c (pkg\paths) : lua_pushlightuserdata(L, (void)tmpnames_key);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)engineKey);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)d);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)metaKey);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)signalKey);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)objectKey);
luaQ_setup in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)qtKey);
luaQ_private_noerr in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)engineKey);
QtLuaEngine::Private::~Private in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)engineKey);
luaQ_pushqt in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)qtKey);
luaQ_buildmetaclass in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)qtKey);
luaQ_buildmetaclass in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)qtKey);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)metaKey);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void)((((size_t)type)<<1)|1));
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)((((size_t)type)<<1)|1));
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)qtKey);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)metaKey);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)mo);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)mo);
luaQ_pushmeta in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)qtKey);
luaQ_pushqt in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)objectKey);
luaQ_pushqt in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)obj);
luaQ_pushqt in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)obj);
QtLuaEngine::Receiver::universal in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)signalKey);
QtLuaEngine::Receiver::universal in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)this);
QtLuaEngine::Private::processQueuedSignals in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)signalKey);
QtLuaEngine::Private::processQueuedSignals in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)this);
QtLuaEngine::Private::processQueuedSignals in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)signalKey);
QtLuaEngine::Private::processQueuedSignals in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)receiver);
luaQ_connect in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)signalKey);
luaQ_connect in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)r);
luaQ_disconnect in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)signalKey);
luaQ_disconnect in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)r);
luaQ_disconnect in qtluaengine.cpp (exe\qtlua\qtlua) : lua_pushlightuserdata(L, (void*)r);

[vielmetti]

The following three PRs illustrate how PowerDNS worked around the lightuserdata issue on arm64, perhaps as a model for the work you are doing. via @Habbie

Where addresses (the address of a type_info) were used as indexes, we replaced those with (semi-human readable) strings (the mangled type names) - ahupowerdns/luawrapper#46

Where addresses needed to be stored for later retrieval, we boxed them in a full userdata - ahupowerdns/luawrapper#45

Where we were using addresses as indexes but really didn't need to, we replaced that with luaL_ref - ahupowerdns/luawrapper#44

openresty/lua-nginx-module#1152 is another discussion of this case.

[vielmetti]

Hello @cdluminate - did you close this issue because it was resolved?

[cdluminate]

@vielmetti No. It's pointless to keep a bug that won't be fixed forever open.

[vielmetti]

Fair enough, please label as "wontfix".

[waruqi]

Here is a workaround implementation, It works fine.

https://github.com/xmake-io/xmake/blob/96b0eae26058482a83f5bbf57bab15a834339464/core/src/xmake/prefix.h#L45-L82