Node OpenCV Addon

I'm glad to finally say I have a good start as far as I can tell for implementing a good OpenCV addon for node js.

Its not a sprint but a marathon, OpenCV is one of the largest, most complex libraries I had to deal with and I've tested many ways to implement it. finally found a sound way to actually do it.

The adventure started some time ago when I attempted to use a few functions to display an augmented reality diffusion tensor file on a marker with a tablet (I got some help from Frank from DSI Studio, Thanks Frank!), it later became a challenge as I learned more and more about C++, Node js and OpenCV.

The first few things I've tried was using ffmpeg api in node js, ffmpeg is written in C, which lacks automatic object lifetime management (constructors/destructors) but contains allocators where you need to explicitly free unused objects and buffers.

I've decided the best way was to try and implement a C++ wrapper on top of these APIs. it was challenging to find the appropriate lifecycle from the documentation but reading the C code helped a lot, eventually I've completed a C++ wrapper which could access most of ffmpeg API with an intuitive API.

The adventure didn't end there, I've used OpenCV Matrix (cv::Mat) as the base container for both image and audio frames which allowed me to explore the OpenCV API with both video and passthrough audio but with a possibility to eventually process the audio as well since its easily accessible.

I've started to explore the OpenCV API and I really liked the idea to manipulate the video frames with Node js, so I decided to implement a few of OpenCV APIs but then I thought how hard would it be to implement the entire OpenCV API in node js, so I went ahead and learned about Node Addons, V8 and NAN.

Many people think that Javascript is not suitable for processing Video/Audio but the truth of the matter is that Javascript is not really processing anything, its just a scripting engine. The real magic stays in C/C++ domain and the delay Node js/v8 adds is negligible when you see how much time compressing a frame or executing canny on it takes, those microseconds do add up, but in terms of percentage, it should be nearly invisible in the total program execution time, besides, this is more for fun than for science.

But... Javascript is not entirely suitable for scripting complex APIs and if you add the number of function overloads possible in OpenCV, it will probably be hell to use not to talk about development, this made me think in the Typescript direction, which can both help the intellisense and make development easier since it enforces some type checking at development time and that could eventually be used to make the API a lot more readable and intuitive.

A few things made developing the API not the most fun thing in the world, for example, the way v8 exposes its internal objects, everything looks like an object, all numbers are actually a double and there is no possibility to overload function implementation internally.

These reasons and more made me think and develop node-overload-resolution project, which addressed most of these problems by allowing me to add overloads to functions, add parameter validations, automatic conversion between v8 data types and C++ data types, automatically execute function implementations in the libuv threadpool as async functions and handling return values and exceptions without explicitly writing a single line of code in the addon itself.

So why am I writing all of this today?
Today the first test passed, not my tests, not all the tests, but the first OpenCV test which I ported to typescript to make sure the API is working properly.

Hopefully its not the last :-)

But let me be completely honest, it is by no way or shape ready for anything, the amount of APIs implemented are very little, even the build at the moment is complex and takes anywhere between 30 minutes to an hour. Since I've created the opencv_ts branch I didn't even check if the code compiles on linux, which it did before, on linux32/64 and even arm and NDK, sorry linux guys, in terms of IDE for C++, I have yet to see anything which comes close to Visual Studio ;-) Tags: node, opencv, v8