Our Take on Image Processing Techniques and the Importance of Expert Approach
Modern image processing technologies, tools, approaches, and techniques certainly demonstrate a “sci-fi-like” level of advancement. We get to seamlessly edit images in intricate ways that let us achieve spectacular visual results that seem to have come straight out of a “Back to the Future” movie. For one thing, one can quite easily create an image with a holographic effect without resorting to any “spaceship technologies”.
All you need is the right device, proper software, and required editing skills. We have managed to achieve quite a few success milestones in terms of mastering progressive image processing via professional hardware. In particular, we are interested in employing in practice and further researching three specific image processing hardware pieces.
NVIDIA Jetson TX2 Module
An AI-powered module-based supercomputer by NVIDIA is a powerhouse of an image processor that allows achieving some extraordinary results and is easily integrated with any PC installation. An 8GB NVIDIA GPU from the Pascal family that runs at 59.7GB/s memory bandwidth lies in the core of the module. The embedded artificial intelligence of this solution does wonders at granting professional-grade camera processing capabilities.
Framos FSM-IMX283C
Sony IMX283 20.2 MPix sensor, this neat-looking module is a real find for professional image processing teams. Combined with one more sensor and other underlying image-processing tech, it allows to conveniently capture hi-def panoramic view shots to further process them into a full-on VR image format or leave as is (in a sort of a fisheye format). This, however, requires certain specialized efforts as well as some trial and error if you are working with this tech stack and looking for ways to achieve the best results (read further).
Evetar E3267A-R3
This is a super-wide 250° fisheye lens that enabled us to capture the surroundings for further 360 image composition. And it features in-depth color aberration controls, HFOV of 200°, IR correction, and full 360 view capacities. This is a great multipurpose lens overall that is best suited for dimensional shots.
We set up this particular hardware stack in order to tackle complex full-view image processing tasks and, eventually, managed to achieve professional output and assets for app development. But before we dive into specifics, let’s figure out what exact assets we can create with this stack.
Progressive Dimensional Formats We Can Provide
There are several major ways the main 360-view shot can be processed, each can be used for specific purposes and grab the onlooker’s attention with a different level of “showiness”.
Monoscopic Motion 360
This is a VR format that is based on displaying two identical images for both eyes side by side through a single channel (hence “monoscopic”). The format requires high-grade image capturing of no less than 4K resolution from the get-go. The “old television” effect of such videos can only be fixed via the highest density of pixels in the source image. Mono 360 video is equirectangular in shape, with a 2:1 aspect ratio.
Stereoscopic 3D Motion 360
Now stereo full-view videos are separated by channels – a channel per eye – in order to enable a 3D effect and provide that immersive depth perception effect we see when we go to watch a 3D movie in the cinema. By slightly shifting the perspective of two channels, shapes become dimensional, and foreground objects as though become separated from the background. The important technical aspect of achieving stereo 3D 360 is that your resolution is cut in half for two channels – this is where increasing the initial mono resolution 2X does the trick.
VR 180 Motion 3D
A new trend in the industry – making immersive VR videos of events and other things that don’t require you to actually turn around and look behind your shoulder. A 180-degree full-frontal view does just fine for numerous purposes out there. For instance, watching live performances or reality shows in VR can be done most conveniently through VR 180 motion 3D.
Our Work with the Image Processing Hardware Stack
The combination of NVIDIA Jetson X2 as a core module with an IMX283 sensor and a super-wide fisheye lens enabled us to start capturing panoramic images that can be further converted into the monoscopic VR 180 format. And employing two IMX283 sensors jointly allows for 360-degree picture capturing. However, this is where we came across some intensive image processing efforts related to the seamless image edges stitching, which we managed and will tell you exactly in the next article.
One of the major challenges we confronted was that the E3267A lens is not directly compatible with the IMX283 sensor, which limited us from using full sensor resolution. This led us to ultimately achieve the 1840×1840 (3.4 of 20.2 MPix) picture size.
In order to get a full-color dimensional image, we needed to supply our IMX283 sensor, which captures raw images at a 10/12-bit color depth within the RG2RGB color pattern, with an appropriate Bayer filter. This enabled us to conduct full-color image reconstruction and further convert the fisheye image into a panoramic view or some other VR image format. Managing all actions via OpenCV and Python, we concluded our experiments with a bunch of finished fisheye and panoramic images.
Further calibrations
Currently, we can clearly see that we need to find a quality-boosting method of some sort because the resulting images lack colors due to the insufficient individual sensor calibration. There is also a slight film of blur which occurs because only a small piece of footage is observed by a viewer in the 360 VR image. We are experimenting on doubling the resolution and making even the smallest pieces of the demonstrated image as vivid and clear as possible.
Final Thoughts
Coming from firsthand experience, we can say that some outstanding, perhaps even market-defining capabilities can be achieved through the right hardware and a proper approach to dealing with it. The tech stack consisting of 3-4 above-mentioned devices can do wonders alone.
Meanwhile, we have already formed our image processing techniques improvement agenda and will be glad to put the expertise we already have into your project. Contact us to discuss details.