The RealSense D435 camera can be readily set up to publish all the image and pointcloud streams via ROS topics. For an in-depth setup, read Integrating the Intel RealSense D435 with ROS. Once this data becomes available in ROS, processing it via custom nodes requires a larger ROS ecosystem for perception. Three popular 2D/3D computer vision libraries will be covered in this post.

Point Cloud Library (PCL) in ROS

The PCL Library is an open-source set of algorithms for processing 3D geometry. It is written in C++ and sufficiently optimised to run algortihms such as segmentation, surface reconstruction, registration, feature estimation, and filtering. To set up PCL in ROS, simply install the perception_pcl ROS package:

• • cd ~/robot_ws/src
  • Clone the repository: git clone -b kinetic-devel https://github.com/ros-perception/perception_pcl.git
  • Build the project: cd ~/robot_ws and then catkin_make
  • There is a dependency on libopenni2 which can be installed using sudo apt-get install libopenni2-dev

OpenCV Library in ROS

The OpenCV Library is an open-source computer vision library designed for computational efficiency. It contains many computer vision algorithms and has a strong global user community. The Using ROS with OpenCV post provides a good overview of how to interface OpenCV with ROS. To install OpenCV in ROS, simply install the vision_opencv package:

• cd ~/robot_ws/src
• Clone the repository: git clone -b kinetic https://github.com/ros-perception/vision_opencv.git
• Build the project: cd ~/robot_ws and then catkin_make

This installs the cv_bridge, image_geometry, and opencv_test packages.

Boost C++ Library

Boost is a collection of C++ libraries for linear algebra, image processing, unit testing and other tasks or structures. To install this library simply  run: sudo apt-get install lib boost-all-dev.

PyTorch Libray in ROS

The PyTorch Library is an open-source Python machine learning library that is popular within the Deep Learning research community. PyTorch supports tensor computation, GPU acceleration, and Deep Neural Networks. There is no official ROS integration so this section  describes a general installation from source.

Installing PyTorch from source is a four step process:

Install Anadonca 5.3 for Linux Python 3.7:

• curl -O https://repo.anaconda.com/archive/Anaconda3-5.3.0-Linux-x86_64.sh
• Run the script: sh Anaconda3-5.3.0-Linux-x86_64.sh. Make sure to select no for the Visual Studio code installation in this step.
• Get access to the conda command by running: source ~/.bashrc

Install CUDA:

To support GPU acceleration in PyTorch, CUDA Toolkit 10  must be installed. For a laptop with the NVIDIA GTX1080 GeForce graphics card, follow the deb install guide:

• Download the file then run: sudo dpkg -i cuda-repo-ubuntu1604-10-0-local-10.0.130-410.48_1.0-1_amd64.deb
• Add the key: sudo apt-key add /var/cuda-repo-/7fa2af80.pub
• Update your packages: sudo apt-get update
• Install CUDA: sudo apt-get install cuda

Install dependencies:

Dependencies can be installed via conda following the Official PyTorch Linux Installation Guide.

• Add the Anaconda path: export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
• conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing
• conda install -c mingfeima mkldnn
• Add LAPACK GPU support: conda install -c pytorch magma-cuda90

Install PyTorch:

The system is finally ready for a PyTorch installation. In your home directory, run these commands:

• Clone the PyTorch repo: git clone --recursive https://github.com/pytorch/pytorch
• Install PyTorch: cd pytorch then python setup.py install
• Install PyTorch vision: conda install torchvision

If you get errors, make sure you’re using a supported version of Python and CUDA. With these libraries installed, it is time to start developing robot perception algorithms.

Updating PyTorch:

To update your PyTorch installation pull the latest version from Github and make sure to also update the submodules:

• • Get the updated repository cd pytorch then git pull origin master
  • Update the submodules git submodule sync then git submodule update --init --recursive
  • Build and install the project python setup.py install