Just tried that build and got no error on my Q X7.
BUT it looks like Opentx is doing something when a new bf lua version is loaded even if it's precompiled. Running it for the first time I'm seeing a huge memory spike. Free memory drops down to 1.4kb. Restarting the radio and everything is back to normal. 32kb free memory.

@mikeller From looking at the console output of the jenkins builds it looks like the script is compiled using lua 5.3. Opentx uses 5.2.2. Could this cause Opentx to recompile the script when a jenkins build is used?

Also there have been some changes since 1.4.1 was released which causes the script to use more memory. There are now two versions. One for the telemetry screen and one for the tools menu.
@ApophysX could you try removing the script from the telemetry screen and try running it from the tools menu instead? It uses less memory than the telemetry version.

Hello klutvott123,
I started SCRIPTS \ TOOLS \ bf.luac and bf.lua, but there is still script syntax error not enouge mamory.

@klutvott123: It could - I am not a specialist in lua versioning. Can you compile the scripts yourself and test this hypothesis? (Or maybe copy the .luac files off one TX, and add them to another one?)

@klutvott123 @mikeller loading the released scripts in the simulator gives the following:

luaLoadScriptFileToState(/SCRIPTS/TELEMETRY/bf.lua, T): loading /SCRIPTS/TELEMETRY/bf.lua
luaLoadScriptFileToState(/SCRIPTS/BF/protocols.lua, bt): loading /SCRIPTS/BF/protocols.luac
-E- luaLoadScriptFileToState(/SCRIPTS/BF/protocols.lua, bt): Error loading script: /SCRIPTS/BF/protocols.luac: incompatible precompiled chunk
	Retrying with /SCRIPTS/BF/protocols.lua

luaDumpState(/SCRIPTS/BF/protocols.luac): Saved bytecode to file.

Which is usually caused by 32/64 bit mismatch issues (although maybe lua version as well?) so it does look like it's being recompiled...

(This was using a fresh download of the released 1.4.1)

@klutvott123, @frozenskys: Trying to start the simulator from companion23 sinks the entire app into a memory hole - I've given up on getting this to work.

Can you please try if the attached build fixes that version:

betaflight-tx-lua-scripts-1.4.1_32bit.zip

@mikeller I'm still seeing the same "incompatible precompiled chunk" using that build... Let me dig further as looking on stackoverflow, lua is very picky about it's bitness, and endianness...

@frozenskys: Would just not supplying the .luac files be a solution?

@mikeller I agree as this is about where I have got to - It looks like we would need to cross compile to ARM because as far as I can work out lua scripts seem to be very picky about how and where they are compiled...

I'm not sure that this would help the initial memory spike during OpenTx compiling the scripts - but if OpenTx is going to re-compile on first run anyway...

Hi,
I'm not sure now, but I think we're talking past each other. With the Betaflight TX Lua Scripts 1.4.1 version my X9D + has no problems. I was referring to the Last Build (# 64), 1 day 18 hours ago from https://ci.betaflight.tech/job/Betaflight%20Lua%20-%20X7,%20X9,%20X9D,%20X9D+,%20X10 ,% 20x12 /

@mikeller , @frozenskys I also did some investigating and have come to the same conclusion. I tried compiling the files in the sim and copying them over to my Q X7. It still recompiles the script. The luac files produced by the sim are identical to the files produced by the radio. Checked them in a hex editor. The memory spike is the same as when just copying over the files directly from src.

Also did some reading in lua documentation. It says that precompiled chunks are not portable across architectures. I'm not sure if that is the case because I get the same files when compiling on my windows pc as I do on my raspberry pi. 😕

@ApophysX I think your radio just doesn't have enough free memory to run the latest build.
Could you try this?
Remove the script from the telemetry screen to prevent it from running when turning on the radio.
Restart the radio.
Hold enter and go into the statistics screen.
Look at "free mem".
How much?

@klutvott123 I've been looking around and there have also been reports on the iNav github of memory issues - I'm wondering if it can be caused by something like flex being included in the firmware or something else at the firmware level that could cause low memory?

Did we add that much more memory use in the changes between 1.4.1 and 1.5.0 ?

@frozenskys Maybe. But flex was available in 2.2.3 also? I see in the companion that flex is still optional. Or maybe you mean ACCESS support?

Memory use has gone up a bit since 1.4.1. The pages use a few KB more than they used to. "Free mem" in opentx is also higher by maybe 6-7KB when using 1.4.1. Tools version uses a little less memory than telemetry version. Probably because there's no background function for one time scripts. I use

local memUse = collectgarbage("count")
lcd.drawText(0, 56, string.format("%.1f", memUse))

at the bottom of run_ui() to monitor total lua memory use in combination with "Free mem".

@klutvott123,
my X9D plus (2015) have 70032b free mem.

After Remove the script from the telemetry screen and reboot my Tarenis and load SCRIPTS \ TOOLS \ bf.luac it starts now.

@ApophysX Good. That should be enough to run the scripts, but it might not be enough for opentx to compile them.

Can you tell us exactly what the error message says? Is it just "script syntax error not enough memory"?
Try deleting all of the .luac files. Then turn on your tx and try running the script again. Does it give you a different error message?

@frozenskys: Kind of sad - to me it looks like lua makes this harder than it needs be. Is there any downside in performance to having the .luac files in the current release (i.e. do we need to do a new release that removes the .luac files ASAP)?

@mikeller no - no downside to shipping the .luac files as far as I can see, as they will just be re-compiled anyway - it's just currently pointless so we should stop doing it.

More pressing is to work out why a few people are seeing memory errors, and seeing if we can shave a few Kb back off the refactored scripts.

@frozenskys I think maybe vtx_defaults.lua can be optimised a bit in terms of memory use. That file is loaded by vtx.lua and will always add to the total memory use. In addition it uses global variables which uses more memory in some cases. If that file could be loaded into a local variable in vtx.lua we could avoid using global variables and it would be unloaded when the vtx page is unloaded.

@klutvott123 good call - I'll take a look at it when I get home tonight. But the data also looks higher for all the scripts after the refactoring - I wonder if that's because it's using two tables for the fields now, or if the removal of the pre_xxx.lua scripts is more to blame? It's a long time since I last had to worry about memory optimisation...

@ApophysX Good. That should be enough to run the scripts, but it might not be enough for opentx to compile them.

Can you tell us exactly what the error message says? Is it just "script syntax error not enough memory"?
Try deleting all of the .luac files. Then turn on your tx and try running the script again. Does it give you a different error message?

@klutvott123, i deleted all * .luac files from the SD of my Taranis (64 elements). Then I started SCRIPTS \ TOOLS \ bf.lua because I get the message "Script syntax error TOOLS / bf.lua: 14 not enough memory"

@frozenskys I would imagine using two tables has a bigger impact than removing the ``pre_xxx.lua```files but I'm not sure.

@ApophysX That means it ran out of memory when compiling the script. Restart your radio and try running the script again. The files it has already compiled won't be compiled again so it will need less memory this time. Then you could try doing the same with the telemetry screen version of the script.
BTW you can run the tools version directly from the tools menu. It's the page right after the SD card page.

@klutvott123, okay it works now, on the Tools menu (page 3 I was not aware, thanks for the info) and in the telemetry screen starts VTX Setup directly.

@ApophysX Nice! Thanks for testing 👍

@mikeller , @frozenskys I have been looking into the precompile situation trying to find solutions.
I wrote a lua script that loads and compiles all the .lua files one by one. Pointed it to the bf lua src folder and ran it in the simulator to have it compile all the files in there. These files are identical to the ones produced having the radio compile them but the timestamps of the .luac files are older than the .lua timestamps. Then I modified build.sh to just copy the contents of src into obj updating all the timestamps in the process. Then it deletes .luac files from src. Alternatively touch could be used to change the timestamps.
Loaded the precompiled script onto my QX7 and it worked. No memory spike.

Here's the current master precompiled if anyone would like to test it. Included the scripts used to make it.
betaflight-tx-lua-scripts.zip

@klutvott123: So this can work around problems that otherwise make it impossible to run the lua scripts on an actual TX?
All in all this solution sounds very kludgy and will probably never lend itself to be integrated with any CI pipeline, which will make it hard to get consistent results.
Do you think it will be possible to improve on this by either:

• using a lua-cross-compiler for ARM or a lua compiler running inside an ARM virtual machine to generate the .luac files; or
• change the scripts so that on first start on a TX all scripts are loaded one by one / compiled / unloaded?

@mikeller I agree. It's not very elegant. Right now it's mostly just to show that it's possible to have the script precompiled and not have opentx compile it again and run out of memory. I found that it's not impossible for it to run even if it ran out of memory when compiling. Just restart the TX and it doesn't recompile the files it compiled last time. Not very good from a users perspective but it works.

I tried compiling on a raspberry pi which uses an ARM cpu using lua 5.2.2 but didn't have any luck with that. I'm starting to wonder if maybe all official lua versions are incompatible with opentx lua. The opentx version has been tampered with.

It's possible to run the compile.lua I supplied directly on the radio. Might also be possible to do this automatically. I'll look into it

Looking at the first 12 bytes of the header of a .luac file compiled on the sim looks like the following:

1B 4C 75 61     LUA Chunk
52              Version 5.2
00              Official Format Version
01              Little Endian
04              Size of int (4 bytes = 32bit)
04              Size of size_t  (32Bit)
04              Size of Instruction (32Bit)
08              Size of lua_Number (64Bit)
00              floating point

The one compiled on jenkins however has a different (size_t) value:

1B 4C 75 61     LUA Chunk
52              Version 5.2
00              Official Format Version
01              Little Endian
04              Size of int (4 bytes = 32bit)
08              Size of size_t  (64Bit)               <-- Oh No!
04              Size of Instruction (32Bit)
08              Size of lua_Number (64Bit)
00              floating point

I wonder if something like a windows or linux port of eLua could be used to cross compile?

@frozenskys: Could it be that this is something that was broken by OpenTX only recently? I remember that way back when we started we tested supplying the .luac files, and back then they were used by the TX without problems.

@mikeller The only change I can see around this (i.e. in the OpenTx lua code) is from 3 years ago, when they moved from double to float but even then I'm not sure that would affect size_t

@mikeller I checked the headers of the 32bit files you provided and they were compiled with lua5.3 so I'm wondering if a 32Bit install of lua5.2 would do the trick? @klutvott123 what do you think?

@frozenskys: That was a 32 bit install of lua 5.3...

@mikeller yes but lua 5.3 - it generates completely different byte code than 5.2 😢

@frozenskys: Try this:

betaflight-tx-lua-scripts_1.4.1_lua_5_2_32bit.zip

Well that's interesting... The header seems to match, but when trying to run on the sim I get the following error

luaLoadScriptFileToState(/SCRIPTS/TELEMETRY/bf.lua, T): loading /SCRIPTS/TELEMETRY/bf.lua
luaLoadScriptFileToState(/SCRIPTS/BF/protocols.lua, bt): loading /SCRIPTS/BF/protocols.luac
-E- luaLoadScriptFileToState(/SCRIPTS/BF/protocols.lua, bt): Error loading script: memory allocation error: block too big
-E- luaLoad(/SCRIPTS/TELEMETRY/bf.lua): Error parsing script (2): /SCRIPTS/TELEMETRY/bf.lua:5: memory allocation error: block too big

Doing a binary compare between that and the one compiled by the sim shows an identical header but subtly different bytecode... 🤔

Digging further it looks like the LTR Patch from eLua was added to OpenTx sometime around 2014 http://www.eluaproject.net/doc/master/en_arch_ltr.html

So i can't see how any normal lua bytecode could work... if the bitness is incorrect it would just recompile, and if the bitness is correct it won't load as it gets STRING and LIGHTUSERDATA types mixed up...

Good work! This further backs up my suspicion that opentx lua might be "incompatible" with regular lua.
So incorrect bitness might have been "saving" us all this time?
Maybe the reason that this seemed to work earlier is that the spike in memory use was less significant than it is now when the script is recompiled.

Interesting. I think going forward we either have to skip building / distributing .luac files that cannot be used, or find a way to run the OpenTX lua compliler from the command line, so this can be scripted.

@raphaelcoeffic: Do you know if this is already possible, or have an idea of how feasible this is?

@klutvott123 it's definitely incompatible:

OpenTx Lua Types

#define LUA_TNONE               (-1)
#define LUA_TNIL                0
#define LUA_TBOOLEAN            1
#define LUA_TROTABLE            2
#define LUA_TLIGHTFUNCTION      3
#define LUA_TLIGHTUSERDATA      4
#define LUA_TNUMBER             5
#define LUA_TSTRING             6
#define LUA_TTABLE              7
#define LUA_TFUNCTION           8
#define LUA_TUSERDATA           9
#define LUA_TTHREAD             10
#define LUA_NUMTAGS             11

Vanilla Lua 5.2.2 Types

#define LUA_TNONE               (-1)
#define LUA_TNIL                0
#define LUA_TBOOLEAN            1
#define LUA_TLIGHTUSERDATA      2
#define LUA_TNUMBER             3
#define LUA_TSTRING             4
#define LUA_TTABLE              5
#define LUA_TFUNCTION           6
#define LUA_TUSERDATA           7
#define LUA_TTHREAD             8
#define LUA_NUMTAGS             9

In theory we could build an LTR patched version of luac that should produce compatible byte code... but then it would need to be kept in sync with the OpenTx version :(

@frozenskys But then we would have to extract the patch from the opentx version? It's using a "modified" version of LTR. Sounds like a nightmare 😄
Having the opentx compiler accessible from the command line like @mikeller suggested would be ideal. Then we would always be in sync with opentx. I'm sure that's functionality that would be very useful for others making lua scripts for opentx too.

@klutvott123 I'm looking at that now - the OpenTx version is basically Lua 5.2.2 with the LTR patch added - however it's also pulling in some debug C++ headers that are OpenTx specific and aren't standard so the included default make file won't build the linux flavour without some magic...

@frozenskys I just had a go at it myself. Applied the patch as it was done in opentx to a clean lua 5.2.2. Now I'm able to compile identical files to the ones compiled by opentx.

@klutvott123: Have you been able to verify that they are accepted in OpenTX?

I think we really need to strike up a conversation with the OpenTX team for this - it seems that on-device compilation is a major contributor to memory overruns, and giving script authors an easy way to automate off-device compilation would be a way around this, unlocking the potential for more complex scripts.

@mikeller Yes, I have done multiple tests to verify that it works. No memory spike after loading the scripts compiled with this patched lua version.
Here it is if you want it: lua-5.2.2-opentx-ltr.zip

@klutvott123 I've been doing some more profiling, and it looks like after compilation 1.5.0 pages use around 2-3k more than the respective 1.4.1 pages when running in the sim. About 1k of this is caused by the new code for the menu, the rest I'm not sure about.

@frozenskys I tried the vtx_defaults/vtx changes I mentioned earlier and it looks like we can maybe get around 2kb back by doing that. Should I make a PR for it and then you can have a look at it?
1kb for the main menu is 1kb well spent 😄

@frozenskys Like this klutvott123@632ab6e
Could you try it and verify if it makes a difference or not?

@klutvott123 I had applied almost exactly the same change myself (only the variable name is different) but I don't see any change in memory (the following are after compilation using your memory used changes to ui.lua):

• 1.4.1 = 38.5
• 1.5.0 global = 41.5
• 1.5.0 local = 41.4

Also if there is an override "{model_name}.lua" it will still be loaded globally and therefore not applied.

@frozenskys Did you check any of the other pages? It doesn't make much difference on the vtx page itself, but on the other pages there should be a reduction in memory use.
For example here's the filter page before and after.

@klutvott123 Ah OK - my misunderstanding - looks like that's working and good find :) ... We just need to fix model specific vtx_tables now, as they are still loaded globally (which will also need a change to the BF configurator code that dumps the lua tables)

So worst case (compiled) memory use after your fix seems to be vtx page running as a telemetry script at 41.5Kb

@frozenskys Now I see how it's intended to work. I just ignored vtx_tables at first.
Something like this maybe?

local vtx_tables = loadScript("/BF/VTX/"..md.name..".lua")
if vtx_tables then
    vtx_tables = vtx_tables()
else
    vtx_tables = assert(loadScript("/BF/VTX/vtx_defaults.lua"))()
end

That would work - I'll need to update the BF exporter as it currently just writes out the bandTable, frequencyTable and frequenciesPerBand (but adding the other two static tables shouldn't be a problem) and also make it local like the defaults.

@frozenskys All right, let's do it #284