Practical Kinect Stereo Calibration for the Highest Accuracy

I’ve been meaning to write up this post for a while, but I’ve been putting it off ūüôā

The release of the Kinect sensor by Microsoft spawned a plethora of research in robotics, computer vision and many other fields. Many of these attempts involved using Kinect for purposes other that what it was originally meant for! That pretty much involves anything other than gesture recognition and skeleton tracking.

Even though Kinect is a very capable device, many people (including us!) don’t get the fact that Kinect was¬†simply not designed for being used as an all-purpose depth and RGB camera. This observation is further bolstered by the fact that the Kinect was not released with a public API, and the¬†official SDK which was released much later was missing a lot of functionality critical¬†to various computer vision applications. For instance, the SDK does not provide a built-in functionality for getting color camera and depth camera calibration values, extrinsics and distortion models. Rather conveniently, the SDK designers simply chose to provide the users¬†with simple functionality, like mapping pixels from world coordinates to RGB coordinates (but no way to do a backproject an image or depth point to ray). Needless to say that in many practical computer vision applications, one needs to constantly re-calibrate the camera in order to minimize the error caused by the mis-calibration of the camera.

Nevertheless, the choice by the SDK designers is understandable: there are proprietary algorithms and methods that are implemented in the Kinect software layer and it may not always be possible to give public access to them. Also, 3rd party open source libraries (such as libfreenect) have tried to reverse-engineer many of the innerworkings of the Kinect sensor and supply the end user with a set of needed functionalities.

With all this, let’s get started! (If you are impatient, feel free to jump to the list of tips I have compiled for you at the end of this post). I have also included PDFs containing checkerboards suitable for printing on large sheets, PVC or aluminum dibond.

Read the rest of this entry »

Computing the Distance Between a 3D Point and a Pl√ľcker Line

In order to solve an optimization problem with the goal of reducing the distance between a bunch of 3D points and lines, I was looking for the correct way of finding the distance between 3D points and a Plucker line representation.

The Plucker line \(L\) passing through two lines \(A\) and \(B\) is defined as \(L = AB^T РBA^T\) (for more details refer to [1]). After a lot of looking, I found that there is a simple method for finding this distance in [2]. A direct quote from the paper:

 

A Plucker line \(L = (n, m)\) is described by a unit vector \(n\) and a
moment \(m\). This line representation allows to conveniently determine
the distance of a 3D point \(X\) to the line

$$d(X, L) = ||X \times n – m||_2$$

where \(\times\) denotes a cross product.

 

[1] Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.

[2] Brox, Thomas, et al. “Combined region and motion-based 3D tracking of rigid and articulated objects.” IEEE Transactions on Pattern Analysis and Machine Intelligence 32.3 (2010): 402-415.

Tracking down BadImageFormatException in C#

I know It’s been quite some time since my last post, but what can I say? I’ve been quite busy ūüôā Here¬†is the latest issue that I got stuck on for a few hours, and I thought I’d share what worked in the end.

I was working on a large C# project that had a lot of dependencies on native C++ DLLs in Visual Studio 2015. I had compiled all the 3rdparty dependencies to DLLs myself and had included them in the C# project. On my main development machine, everything was working fine, but I couldn’t get the project to work on another machine with a clean Windows 10 install. The project would compile fine but would fail to run with BadImageFormatException.

From prior experience, I knew this error had to do with a mismatch between dependency architectures somewhere (e.g. 32 bit binary is trying to load a 64 bit binary or vice versa). However, this time I was so sure that wasn’t the case because I had built almost all the DLL dependencies myself and I knew that everything was built in x64 mode. Also, it was working fine on my development machine.

After a few hours of frustration and using the Fuslogvw.exe¬†tool without obtaining much more insight about the problem, I decided to use Dependency Walker to see what’s happening. This great little tool did magic! I noticed that one of the dependencies that I hadn’t built myself was actually a 32-bit binary. The reason this was not causing an issue on my development machine was that as bad luck would have it, the 64-bit version of that dependency was in my PATH variable on that machine, so my C# project would load up the correct DLL. However, on other machines with a clean Windows installation, this wasn’t the case and because the only found binary was the 32-bit version, I would receive that dreaded BadImageFormatException.

librealsense with ARM support

Recently, I was able to successfully use RealSense R200 on my NVIDIA Jetson TX1 with librealsense. I had to replace some SSSE3 instructions in the code to get it to compile under ARM. I created a fork of librealsense on Github with all the changes I made. Check it out here: ttps://github.com/Maghoumi/librealsense

R200 works at 60 FPS with my Jetson TX1 flawlessly! ūüôā

Create Bootable USB Flash Drive from ISO Image (with UEFI Support)

Update:¬†If you’re looking for Windows 10 UEFI installation, take a look at the¬†addendum at the end of the post!

Although there are a lot of applications for creating a bootable flash drive using an ISO image (such as UNetBootin), not many of them support the creation of a bootable flash drive that can be used for installing the operating system in UEFI mode (I’ve never succeeded with UNetBootin personally!).

There is a comprehensive guide about installing Linux in UEFI mode that details all the¬†do’s and don’ts. Most of us are already familiar with all the necessary steps but the creation of a bootable UEFI compatible flash drive from an ISO file ūüėÄ .¬†There are two ways to create a UEFI compatible flash drive:

TL;DR

There is a great application called rufus that does the trick for you. You can find it here. The good thing about this app is that the USB flash drive will end up with a single partition and is still usable in Windows.

TL;WR

(That’s Too Long; Wanna Read; for the¬†lack of a better acronym!)

The other possibility is to use the good ol’ Linux utility disk dump ( dd). dd It should be already available under any Linux installation. It is also available under Windows if you have Cygwin installed. The procedure is as follows.

WARNING: You will loose all the data on your flash drive.

NOTE: If you’re using Cygwin, make sure to run Cygwin as Administrator (right click, run as Administrator).

  1. Insert the flash drive and determine what device it has been mapped to by running.
    cat /proc/partitionsThis should output a list of partitions in /dev. The devices and the partitions are numbered. Since you want to work directly with the whole drive, ignore the numbers at the end. In my case, I found out that my flash drive was mapped to /dev/sde (by observing two entries in the list: /dev/sde and /dev/sde1).¬†Make sure you select the correct partition otherwise you’ll ruin another storage device.
  2. Dump the ISO file to the device you noted in step 1:
    dd if=/path/to/your/image.iso of=[the device above] bs=4MIn my case it was:
    dd if=~/image.iso of=/dev/sde bs=4M

This will start dumping the image onto the flash drive. You won’t get any output from the command while the operation is in progress. To check the progress, you need to open another terminal (command) window, determine the PID of dd process by running ps -e (which will give you PID), and then running the command kill -USR1 [PID]; sleep 1; to see the output of dd in the original terminal (command) window.

After the operation has completed, you can boot the flash drive in UEFI mode. The flash drive is booted in UEFI mode if the output of the following command is a list of files:

ls /sys/firmware/efi

Hopefully, the next time I want to install a Linux distro in UEFI mode, I wouldn’t have to Google for two hours.

Addendum

Unsurprisingly, rufust can also be used for Windows 10 UEFI installation. Unfortunately, I didn’t have much luck with Microsoft’s¬†media creation tool. I used it at first, yet I was getting the black screen which says the installation cannot continue (wish I had taken a note of what it said exactly).

Nevertheless, installation with rufus went smoothly. Here’s a direct link to Windows 10 ISO¬†(link courtesy of Reddit). Grab the image, run rufus, select GPT Partition Scheme for UEFI, file system MUST¬†be FAT32 (I chose the block size of 8192, anything other than FAT32 did not boot for me). Click start and you’re golden ūüôā I also had CSM enabled in my bios settings. Not sure if that really mattered though!

UPDATE: I noticed that the above settings had to be slightly different if the goal was an MBR/BIOS installation. What I had to do was to select “MBR Partition Scheme for BIOS or UEFI-CSM“, and the file system must have been¬†NTFS. Also, I had to completely disable secure boot in the BIOS settings to get the flash drive too boot up.

 

Align Depth and Color Frames – Depth and RGB Registration

Sometimes it is necessary to create a point cloud from a given depth and color (RGB) frame. This is especially the case when a scene is captured using depth cameras such as Kinect. The process of aligning the depth and the RGB frame is called “registration” and it is very easy to do (and the algorithm’s pseudo-code is surprisingly hard to find with a simple Google search! ūüėÄ )

To perform registration, you would need 4 pieces of information:

  1. The depth camera intrinsics:
    1. Focal lengths fxd and fyd (in pixel units)
    2. Optical centers (sometimes called image centers) Cxd and Cyd
  2. The RGB camera intrinsics:
    1. Focal lengths fxrgb and fyrgb (in pixel units)
    2. Optical centers (sometimes called image centers) Cxrgb and Cyrgb
  3. The extrinsics relating the depth camera to the¬†RGB camera. This is a 4×4 matrix containing rotation and translation values.
  4. (Obviously) the depth and the RGB frames. Note that they do not have to have the same resolution. Applying the intrinsics takes care of the resolution issue. Using camera’s such as Kinect, the depth values should usually be in meters (the unit of the depth values is very important as using incorrect units will result in a registration in which the colors and the depth values are off and are clearly misaligned).
    Also, note that some data sets apply a scale and a bias to the depth values in the depth frame. Make sure to account for this scaling and offsetting before proceeding. In order words, make sure there are no scales applied to the depth values of your depth frame.

Let depthData contain the depth frame and rgbData contain the RGB frame. The pseudo-code for registration in MATLAB is as follows:

A few things to note here:

  1. The indices x and y in the second group of for loops may be invalid which indicates that the obtained RGB pixel is not visible to the RGB camera.
  2. Some kind of interpolation may be necessary when using x and y. I just did rounding.
  3. This code can be readily used with savepcd function to save the point cloud into a PCL compatible format.

The registration formulas were obtained from the paper “On-line Incremental 3D¬†Human Body Reconstruction for HMI¬†or AR¬†Applications” by Almeida et al (2011).¬†The same formulas can be found¬†here. Hope this helps ūüôā

 

C++ Function in Header throws Linker “already defined” Errors

If you define a function in the global namespace in a C++ header file and encounter linker errors (complaining about the function already defined elsewhere), there’s a simple fix! Simply mark the function as inline. This will prevent the duplication of the function in other source files.

Note that using inclusion guards does not solve this problem and you must define the function as inline.

CGAL Point in Polyhedron Algorithm

The “point in polygon” or “point in polyhedron” is a classic computer graphics problem. The goal is to determine whether a given point is inside a polygon (in 2D) or a polyhedron (in 3D).

One solution to the problem is shooting a ray originating from the said point to an arbitrary direction and determine the number of intersections of the ray with the polygon or polyhedron. If the ray intersects the shape an odd number of times, then the point is inside the shape. Otherwise it is outside the shape.

There are problems associated with this approach. An important edge case is when the point is on the surface of the shape. Also, it is commonly advisable to shoot multiple rays instead of just one, and then do a majority voting.

Luckily, in CGAL the point in polyhedron test is very simple and you don’t have to worry about the edge cases! You can do the test using¬†the Side_of_triangle_mesh class. A code snippet is shown below:

 

You can also determine whether the point is on the surface of the polyhedron or not.

Undo a Git Commit on GitHub

In case you’ve pushed an unwanted commit to GitHub (or any upstream Git repository), you can simply undo it. To do so, move the HEAD to the commit that you want to undo to and then run the following command:

git push -f origin HEAD^:master

Prevent Unity 3D Text from Always Appearing on Top while Maintaining Rich-Text Support

The TextMesh component in Unity has an annoyance which causes the text that it displays be rendered always on top. This is useful if you want your GUI elements to always appear on top but if you want to have a 3D text attached to a 3D object, this creates the weirdest results!

There’s already an entry in Unity’s wiki page detailing how the issue can be worked around using a custom shader. This YouTube video shows the application of the mentioned method (it’s quite useful even though the guy is speaking Russian!). Also, if you are not sure how a texture can be built using a font, follow the steps described in this awesome blog post.

The issue with the above method is that it messes with the rich text support of TextMesh. Essentially, all text will be rendered with the specified color of the created material. I found a better custom shader in the Unity 3D forum! This solved my z-depth issues.

Older posts «