## OpenGL not Found, “cannot find -lGL”, and Other Issues with NVIDIA Drivers

Under Ubuntu,  sometimes when the NVIDIA drivers are installed, CMake cannot properly find OpenGL (specifically libGL.so). As a result, linking issues may occur.

The easiest fix for this is to first create a symlink of nvidia-current to nvidia-xxx where xxx is the version of the current NVIDIA driver installed (e.g.  nvidia-387) . Then create a symlink from /usr/lib/libGL.so to /usr/lib/nvidia-current/libGL.so.

The command would be:

Note that having nvidia-current symlink is generally a good idea as it can prevent other linker errors when it comes to finding NVIDIA-specific libraries. Also, remember to update this symlink whenever NVIDIA drivers are also updated.

## Regional and Locale Settings Affects Parsing Decimal Strings in C#

Yesterday, someone who was trying to use Jackknife, our gesture recognition library, reported that he couldn’t reproduce our results with our C# code. Naturally, we started looking into the issue.

Strangely enough, the code worked fine for me (both Debug and Release configurations). I then built the code under multiple versions of Visual Studio, tried both Windows 8 and Windows 10, and even built the code on Mono under Ubuntu. Everything was working as expected. He even sent me his compiled binaries so that I could try his build, yet to my surprise I was still getting correct outputs on all my machines.

Since our C++ code was working fine for him, I knew the issue was connected to .NET somehow. While trying to figure out what was causing this mind-boggling issue, I noticed that he was in Europe and I got this crazy idea that his operating system’s locale settings were interfering with something somewhere. As a last resort, I changed the regional settings on my computer to his region and voila! I was able to reproduce his problem!!

It turned out that in his region, decimal numbers are separated by commas, rather than periods, which is common in the US. Consequently, in that region the decimal number 123.45 is written as 123,45. In our tool, we were manually parsing CSV files. Since we hadn’t thought of issues like this, we were naively calling double.Parse() on strings containing decimals. Therefore, .NET was using the operating system’s locale settings to parse the string, giving completely different results than the ones we expected.

One easy hack to fix this issue was to force the locale of the current thread to “en-US” locale. The code excerpt below does exactly that:

In our C++ code, however, we were using std::stod which does not care about locale settings, so it was working fine.

## Orthonormalize a Rotation Matrix

If you use a 3×3 R matrix to store the result of the multiplication of a series of rotation transformations, it could be the case that sometimes you end up with a matrix that is not orthonormal (i.e. det(R) != 1 and R.R' != eye).

In such cases, you need to re-orthonormalize the rotation matrix, which can be done in either of the two following ways:

1. Use the SVD decomposition as follows (MATLAB syntax):
2. Alternatively, you can express the rotation matrix as a quaternion, normalize the quaternion, then convert the quaternion back to the rotation matrix form. In MATLAB, you could do this:

Note that the above syntax requires MATLAB’s robotics toolbox.

## Visual Studio Setup Blocked or Can’t Uninstall

For some reason a Visual Studio 2015 Community Edition installation I had on one of my machines was experiencing problems: solution files were not loading properly, and what’s worse is that installation instance had gone totally missing from “Uninstall a Program” in Windows Control Panel. Also, every time I tried running the setup file vs_community.exe from the ISO image, I would get an error saying “The product version that you are trying to set up is earlier than the version already installed on this computer”.

What I ended up doing was to run the setup file with the two following switches:

vs_community.exe /uninstall /force

Luckily, it went forward with uninstalling the instance I had on my machine.

## Practical Kinect Stereo Calibration for the Highest Accuracy

I’ve been meaning to write up this post for a while, but I’ve been putting it off 🙂

The release of the Kinect sensor by Microsoft spawned a plethora of research in robotics, computer vision and many other fields. Many of these attempts involved using Kinect for purposes other that what it was originally meant for! That pretty much involves anything other than gesture recognition and skeleton tracking.

Even though Kinect is a very capable device, many people (including us!) don’t get the fact that Kinect was simply not designed for being used as an all-purpose depth and RGB camera. This observation is further bolstered by the fact that the Kinect was not released with a public API, and the official SDK which was released much later was missing a lot of functionality critical to various computer vision applications. For instance, the SDK does not provide a built-in functionality for getting color camera and depth camera calibration values, extrinsics and distortion models. Rather conveniently, the SDK designers simply chose to provide the users with simple functionality, like mapping pixels from world coordinates to RGB coordinates (but no way to do a backproject an image or depth point to ray). Needless to say that in many practical computer vision applications, one needs to constantly re-calibrate the camera in order to minimize the error caused by the mis-calibration of the camera.

Nevertheless, the choice by the SDK designers is understandable: there are proprietary algorithms and methods that are implemented in the Kinect software layer and it may not always be possible to give public access to them. Also, 3rd party open source libraries (such as libfreenect) have tried to reverse-engineer many of the innerworkings of the Kinect sensor and supply the end user with a set of needed functionalities.

With all this, let’s get started! (If you are impatient, feel free to jump to the list of tips I have compiled for you at the end of this post). I have also included PDFs containing checkerboards suitable for printing on large sheets, PVC or aluminum dibond.

## Computing the Distance Between a 3D Point and a Plücker Line

In order to solve an optimization problem with the goal of reducing the distance between a bunch of 3D points and lines, I was looking for the correct way of finding the distance between 3D points and a Plucker line representation.

The Plucker line $$L$$ passing through two lines $$A$$ and $$B$$ is defined as $$L = AB^T – BA^T$$ (for more details refer to [1]). After a lot of looking, I found that there is a simple method for finding this distance in [2]. A direct quote from the paper:

A Plucker line $$L = (n, m)$$ is described by a unit vector $$n$$ and a
moment $$m$$. This line representation allows to conveniently determine
the distance of a 3D point $$X$$ to the line

$$d(X, L) = ||X \times n – m||_2$$

where $$\times$$ denotes a cross product.

[1] Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.

[2] Brox, Thomas, et al. “Combined region and motion-based 3D tracking of rigid and articulated objects.” IEEE Transactions on Pattern Analysis and Machine Intelligence 32.3 (2010): 402-415.

## Tracking down BadImageFormatException in C#

I know It’s been quite some time since my last post, but what can I say? I’ve been quite busy 🙂 Here is the latest issue that I got stuck on for a few hours, and I thought I’d share what worked in the end.

I was working on a large C# project that had a lot of dependencies on native C++ DLLs in Visual Studio 2015. I had compiled all the 3rdparty dependencies to DLLs myself and had included them in the C# project. On my main development machine, everything was working fine, but I couldn’t get the project to work on another machine with a clean Windows 10 install. The project would compile fine but would fail to run with BadImageFormatException.

From prior experience, I knew this error had to do with a mismatch between dependency architectures somewhere (e.g. 32 bit binary is trying to load a 64 bit binary or vice versa). However, this time I was so sure that wasn’t the case because I had built almost all the DLL dependencies myself and I knew that everything was built in x64 mode. Also, it was working fine on my development machine.

After a few hours of frustration and using the Fuslogvw.exe tool without obtaining much more insight about the problem, I decided to use Dependency Walker to see what’s happening. This great little tool did magic! I noticed that one of the dependencies that I hadn’t built myself was actually a 32-bit binary. The reason this was not causing an issue on my development machine was that as bad luck would have it, the 64-bit version of that dependency was in my PATH variable on that machine, so my C# project would load up the correct DLL. However, on other machines with a clean Windows installation, this wasn’t the case and because the only found binary was the 32-bit version, I would receive that dreaded BadImageFormatException.

## librealsense with ARM support

Recently, I was able to successfully use RealSense R200 on my NVIDIA Jetson TX1 with librealsense. I had to replace some SSSE3 instructions in the code to get it to compile under ARM. I created a fork of librealsense on Github with all the changes I made. Check it out here: ttps://github.com/Maghoumi/librealsense

R200 works at 60 FPS with my Jetson TX1 flawlessly! 🙂

## Create Bootable USB Flash Drive from ISO Image (with UEFI Support)

Update: If you’re looking for Windows 10 UEFI installation, take a look at the addendum at the end of the post!

Although there are a lot of applications for creating a bootable flash drive using an ISO image (such as UNetBootin), not many of them support the creation of a bootable flash drive that can be used for installing the operating system in UEFI mode (I’ve never succeeded with UNetBootin personally!).

There is a comprehensive guide about installing Linux in UEFI mode that details all the do’s and don’ts. Most of us are already familiar with all the necessary steps but the creation of a bootable UEFI compatible flash drive from an ISO file 😀 . There are two ways to create a UEFI compatible flash drive:

## TL;DR

There is a great application called rufus that does the trick for you. You can find it here. The good thing about this app is that the USB flash drive will end up with a single partition and is still usable in Windows.

## TL;WR

(That’s Too Long; Wanna Read; for the lack of a better acronym!)

The other possibility is to use the good ol’ Linux utility disk dump ( dd). dd It should be already available under any Linux installation. It is also available under Windows if you have Cygwin installed. The procedure is as follows.

WARNING: You will loose all the data on your flash drive.

NOTE: If you’re using Cygwin, make sure to run Cygwin as Administrator (right click, run as Administrator).

1. Insert the flash drive and determine what device it has been mapped to by running.
cat /proc/partitionsThis should output a list of partitions in /dev. The devices and the partitions are numbered. Since you want to work directly with the whole drive, ignore the numbers at the end. In my case, I found out that my flash drive was mapped to /dev/sde (by observing two entries in the list: /dev/sde and /dev/sde1). Make sure you select the correct partition otherwise you’ll ruin another storage device.
2. Dump the ISO file to the device you noted in step 1:
dd if=/path/to/your/image.iso of=[the device above] bs=4MIn my case it was:
dd if=~/image.iso of=/dev/sde bs=4M

This will start dumping the image onto the flash drive. You won’t get any output from the command while the operation is in progress. To check the progress, you need to open another terminal (command) window, determine the PID of dd process by running ps -e (which will give you PID), and then running the command kill -USR1 [PID]; sleep 1; to see the output of dd in the original terminal (command) window.

After the operation has completed, you can boot the flash drive in UEFI mode. The flash drive is booted in UEFI mode if the output of the following command is a list of files:

ls /sys/firmware/efi

Hopefully, the next time I want to install a Linux distro in UEFI mode, I wouldn’t have to Google for two hours.

Unsurprisingly, rufust can also be used for Windows 10 UEFI installation. Unfortunately, I didn’t have much luck with Microsoft’s media creation tool. I used it at first, yet I was getting the black screen which says the installation cannot continue (wish I had taken a note of what it said exactly).

Nevertheless, installation with rufus went smoothly. Here’s a direct link to Windows 10 ISO (link courtesy of Reddit). Grab the image, run rufus, select GPT Partition Scheme for UEFI, file system MUST be FAT32 (I chose the block size of 8192, anything other than FAT32 did not boot for me). Click start and you’re golden 🙂 I also had CSM enabled in my bios settings. Not sure if that really mattered though!

UPDATE: I noticed that the above settings had to be slightly different if the goal was an MBR/BIOS installation. What I had to do was to select “MBR Partition Scheme for BIOS or UEFI-CSM“, and the file system must have been NTFS. Also, I had to completely disable secure boot in the BIOS settings to get the flash drive too boot up.

## Align Depth and Color Frames – Depth and RGB Registration

Sometimes it is necessary to create a point cloud from a given depth and color (RGB) frame. This is especially the case when a scene is captured using depth cameras such as Kinect. The process of aligning the depth and the RGB frame is called “registration” and it is very easy to do (and the algorithm’s pseudo-code is surprisingly hard to find with a simple Google search! 😀 )

To perform registration, you would need 4 pieces of information:

1. The depth camera intrinsics:
1. Focal lengths fxd and fyd (in pixel units)
2. Optical centers (sometimes called image centers) Cxd and Cyd
2. The RGB camera intrinsics:
1. Focal lengths fxrgb and fyrgb (in pixel units)
2. Optical centers (sometimes called image centers) Cxrgb and Cyrgb
3. The extrinsics relating the depth camera to the RGB camera. This is a 4×4 matrix containing rotation and translation values.
4. (Obviously) the depth and the RGB frames. Note that they do not have to have the same resolution. Applying the intrinsics takes care of the resolution issue. Using camera’s such as Kinect, the depth values should usually be in meters (the unit of the depth values is very important as using incorrect units will result in a registration in which the colors and the depth values are off and are clearly misaligned).
Also, note that some data sets apply a scale and a bias to the depth values in the depth frame. Make sure to account for this scaling and offsetting before proceeding. In order words, make sure there are no scales applied to the depth values of your depth frame.

Let depthData contain the depth frame and rgbData contain the RGB frame. The pseudo-code for registration in MATLAB is as follows:

A few things to note here:

1. The indices x and y in the second group of for loops may be invalid which indicates that the obtained RGB pixel is not visible to the RGB camera.
2. Some kind of interpolation may be necessary when using x and y. I just did rounding.
3. This code can be readily used with savepcd function to save the point cloud into a PCL compatible format.

The registration formulas were obtained from the paper “On-line Incremental 3D Human Body Reconstruction for HMI or AR Applications” by Almeida et al (2011). The same formulas can be found here. Hope this helps 🙂