Lua is an embedded extensible extension language with a C API. In the real world - Lua can be extended using C API ‘plugins’ to expand its feature set, and extend the hosts’ functionality and integration with external tools & processes.
To achieve a Moho Plugin system - Moho would need to operate with the C/C++ code (Moho application) controlling the Lua script library, and the Lua code then controlling a C library (plugin code). i.e. C -{api}- Lua -{api}- C. In each case the same Lua C API is used, and the plugin/s could even invoke Lua script with further C plugins … etc. etc.
A real-world example of the use of the C API is the integrated Lua math library [ math.sin(), math.sqrt(), math.pi etc. ] which is written entirely as a C API add-on to the Lua core. table, string, io and os are also API add-on libraries.
The following C API code demonstrates coding examples (2 ‘add_num’ functions) - which simply add 2 numbers together:
Code: Select all
#include "lua.h" // core Lua API
#include "lauxlib.h" // auxiliary helper library (also a Lua core C API add-on)
// API accessible functions are defined with 1 parameter lua_State*, and 1 return int (# of return params)
// Function 'add_num' - with the API Lua State parameter L (which houses the Stack)
// The Stack is the data store used to exchange values between Lua and C
// Simple 'add_num' example
int add_num(lua_State *L)
{
lua_arith (L, LUA_OPADD); // pops 2 top values from Stack, adds them and pushes the result back onto Stack
return 1; // return count of 'returned' values @ top of Stack
}
// More complex 'add_num' example - demonstrating direct manipulation of the Stack
__declspec(dllexport) int add_num(lua_State *L) // '__declspec(dllexport)' windows only
{
lua_Number sum = luaL_checknumber(L, -2); // assign validated number (@top of Stack -1) to 'sum'
sum += luaL_checknumber(L, -1); // get validated number (@top of Stack) & add to 'sum'
lua_pushnumber(L, sum); // push 'sum' onto top of Stack
return 1; // return count of 'returned' values @ top of Stack. Lua discards others
}In Lua script a simple require() or package.loadlib() instruction will incorporate our new add_num function:
Code: Select all
local AddNum = package.loadlib(<path to my_plugin>, "add_num")
print(AddNum(5, 7))
⇒ 12So why do we need a Plugin API when we already have Lua scripting?
Better, Stronger … Faster?On the plus side Moho Plugins could offer:
- Extended functionality beyond that which can be realised with Moho and Lua alone
- Super fast processing, with tight integration with Lua, C and Moho
- A vast array of C/C++ source code, know-how & lineage to exploit
- More seamless interfacing when accessing external tools, frameworks, languages & services
- More difficult to create & maintain. Low-level compiled coding == higher learning curve and slower more complicated time-consuming development cycles
- Less portable & more breakable. Greater need to develop & account for compatibility issues related to host platforms, OS versions and Lua binary & script versions
Case Study – “The need for speed”
Moho’s Lua scripting capability is fantastic, but just occasionally it is unable to deliver on speed, functionality or interoperability. A case in point would be: base64 encoding. Recently I had a requirement to base64 encode some images from within Moho. So I found some pure Lua code to complete this task and this worked fine… until the image source got into the 10’s of Megabytes - and the script started to ‘hang’ for a few minutes. So I sought a faster more optimal solution… and concluded that some of the best alternatives to pure Lua scripting might be:
- Call an external executable. This would work (current solution) but has a crude clunky, dos-popping, limited feedback, non-integrated interface
- Lost Marble provide new Base64 functions. A perfectly optimal integrated solution, but hugely dependent on LMs willingness and their development commitments & priorities
- Roll your own plugin via the C API. Another optimally performant integrated solution, dependent on own / community / free-market developer resources, but with a most-likely quick(er) turnaround. Naturally also dependent on a working Moho Plugin API
The simple answer would be conflicts and compatibility. The more involved answer would be …
#1 - Virtual Machine conflicts
With Lua - each running instance executes byte-code inside a conceptual Virtual Machine. In the current implementation of Moho we have the Lua 5.2 script engine, and when Moho Lua 5.2 is used to invoke a child C API instance, the child is run in a separate or duplicate VM. For Lua 5.2 this is an unsupported configuration which can lead to memory corruption when one VM messes with the other VMs’ state.
Lua 5.2 normally detects duplicate VM scenarios and prevents duplicate VMs by issuing a ‘Multiple Lua VMs detected’ error. That prevention is great, but that prevention also halts the client VM and the calling Lua script. For the purpose of testing with current Moho - multiple Lua 5.2 VMs can minimally coexist - but only after a few tweaks and with limited data exchange and function access.
In Lua 5.2 this VM conflict can be avoided if both Host and Client Lua processes use the same VM. This requires the client to bind to and call into the host VM. This is not however currently possible with Moho - as Moho does not expose the Lua C API.
VM conflict can also be avoided by using Lua 5.4 – which states “The Lua core should work correctly with libraries using their own static copies of the same core, so there is no need to check whether they are using the same address space”. Apparently 5.4 does not have a shared access Global state. Instead - all the VM data is held inside the Lua Local State structure (which is created per client Initialization) - so there is no longer a VM conflict.
#2 - Lua version compatibility
Whilst most generic Lua script code can run on many versions of Lua with a few modifications; The ABI (Application Binary Interface) of Lua major.minor versions are not compatible. i.e. Lua 5.2.x is not binary compatible with 5.1.x nor 5.4.x. etc. So a plugin built for 5.2 would not run with a 5.3 host. We also need to take into consideration the host platform operating system. i.e. Windows / macOS.
Since Lua C API Plugins require binary compatibility, we would need to build the API Plugins at a correct/current version and hope that the Lua version embedded within Moho does not change for a while. Fortunately - Moho’s Lua version is updated infrequently. And unfortunately - Moho’s Lua version is updated infrequently. So we have great Lua version stability - but we may also not be able to exploit the features of later and greater versions - for a while …
Desperately Seeking Solutions
So lets look at what we might need to do in order to facilitate a working Moho Plugin API.
1) Current Moho 12.x / 13.5 (Lua 5.2): Proof of concept ONLY
Any Plugin API deployed in the current Moho application environment should be considered to be a volatile hack, as it is neither a stable nor supported configuration. Nonetheless - in order to create a proof-of-concept we can: match Lua versions, sneak past the multiple Lua VMs test, interact with the Lua C API, invoke custom functions (with passed and received data: numerical, string [in/out] & table [out]) and … experience some Whoop Whoop Eureka moments!
In order to achieve this hack we must:
- Avoid using pre-built Lua binaries/libs lua52.dll/so - as they will detect & error on multiple Lua VMs
- Download latest Moho matching Lua source code from https://www.lua.org/ftp/ (currently: Lua 5.2.4)
- Build Lua shared library from source, after editing lauxlib.c to bypass ‘multiple Lua VMs’ check **
- and build specific targets for Windows (dll) or macOS (so). macOS .dylibs can simply be renamed .so
- Either incorporate the build into our plugin code or use the built Lua shared library as a dependency
At the moment, testing suggests that simply bypassing the duplicate VM checks can result in one Lua VM writing over or invalidating the shared memory / state of the other (possibly related to conflicting memory management) - leading to inconsistent garbled results or a Moho crash. This is most evident when we try to return complex structures (functions, userdata & tables), access Global environment (_ENV / _G) items or excessively manipulate the passed-in State & Stack.
Accessing passed variables of any data type in an API function does not appear to be a problem. However, returning structures and data to Moho can cause Lua corruption issues. In order to minimize/avoid corruption, we should:
- avoid using ‘require’ to load modules with multiple functions, properties & entry points
- load / access single functions via ‘package.loadlib’ (each will be granted its own local State)
- potentially use file I/O for non-simple variable and data structure return & transfer
- avoid accessing Global variables, functions and data via the API
2) Future Moho: LM ship Moho with update to Lua 5.4
A preferred solution would be for Lost Marble to upgrade the embedded Lua version to 5.4. Lua 5.4 is multi-VM friendly, thus eliminating the multiple Lua VMs compatibility issue. Plugins would still need to run a matching Lua binary version and be updated to maintain version parity if/when Moho’s Lua version is next updated.
Naturally - updating the Lua version to 5.4 may also necessitate some refactoring and updates to existing Lua scripts.
3) Current/Future Moho: LM expose the Lua C API
If we are unable to upgrade to Lua 5.4 (any time soon), it may be possible that Lost Marble could provide access to the Moho Lua C API via shared DLL or other mechanism, so that C API Plugins can bind to the same main Lua Virtual Machine as used by Moho - eliminating the multiple Lua VMs compatibility issue. Plugins would need to be updated to maintain binary version parity if/when the Moho Lua version is updated.
--------------------------------------------------------------------------------
What Next?
In Part #2 - I will provide dll/so binaries, code and instructions on how to build and/or run working prototypes.
Feedback, Comments & Corrections are naturally welcome.
Further Reading
- Lua 5.2 Reference Manual: https://www.lua.org/manual/5.2/manual.html
- Programming in Lua (Overview of the C API): https://www.lua.org/pil/24.html
- Binding Code to Lua: http://lua-users.org/wiki/BindingCodeToLua
Glossary
Some hopefully useful info on some key component terms …
Environment (_ENV) - The global scope table (since Lua 5.2). By default everything that is considered to be at global scope in Lua is in the _ENV table, and when we execute statements like: print, we are really executing _ENV.print(). It is initialised to a default Global Environment, but you can set your own environment by assigning _ENV. i.e. _ENV = {} (only affects your own code). If you did that assignment then following functions like print will stop working (be inaccessible) as they are no longer in the new scope. Groups of functions can be set to share a private ‘global’ environment.
Global (_G) - _G is a global variable set to point to _ENV by default. _ENV._G ==> _ENV. It can be re-assigned just like _ENV, in fact it can potentially be reassigned to any type or value - as Lua does not use _G. 3rd party scripts may however use it.
Global State - Sometimes referred to as global state machine - relates to the operation of the Lua Virtual Machine. It is a set of structures, processes and values defined & utilised for garbage collection, memory and state management. From the perspective of Lua code, the global_State structure is completely inaccessible as we don’t need to access to it. It is however crucially important and manages globally unique state information in Lua.
Lua State - A unique data structure assigned to each Lua processing thread, which holds all the info about that running/runnable thread. Each Lua State also has a pointer to Global State and to a ‘global’ environment (_ENV). The State is passed (by reference) between Lua and C, and also between C API process calls. The State also houses a Stack.
Lua Thread - A line of execution with its own Stack, local variables and potentially ‘global’ environment. A Plugin would run in its own thread. Lua is single threaded, so only one thread can execute at any given time. Lua also has a mechanism called coroutines that allow threads to yield & resume processing - emulating multi-threading (executing multiple processes concurrently).
Lua Virtual Machine - Lua runs programs by first compiling them into instructions (bytecode & opcodes), then interprets and executes those instructions with a virtual processor, whilst ensuring correct state and memory management. The entire mechanism for completing these functions is the Virtual Machine.
The Stack - An abstract stacked data store used to exchange values between Lua and C. It can be thought of as a stack of pizza boxes. Each box in this stack can hold any Lua value or nothing. Whenever you (your C app) needs a value from Lua (such as a parameter or the value of a global variable), Lua has to push the required value onto the stack - so that your app can pop it off and/or use it later. Whenever you want to pass values from C to Lua, you first have to push the values onto the stack, so that Lua can pop them off.
Lua manipulates this stack in a strict LIFO order (Last In, First Out; via the top). Lua can only change the topmost part of the stack. Whereas your plugin C code can inspect any element inside the stack and insert and delete elements in any arbitrary position. The bottom of the stack has index 1, and the top is ‘no. of elements’ or -1. p.s. In Lua script you never interact with the stack directly - it is all done on your behalf.
