Carto

Carto is a point cloud acquisition, calibration and fusion tool developped by the innovation department of the Society for Arts and Technologies.

Carto allows you to connect to multiple volumetric sensors and to calibrate their point clouds in order to create a coherent cartography of a space. The resulting fused point cloud can then be sent in real time to other software as orthographic views or as a raw data.

License

Carto is distributed under the AGPL-3.0 license

Installation

TODO: Mettre à jour lorsque des builds seront réellement disponibles

Even though Carto is distributed under a libre license, binaries are only distributed to Suite SAT subscribers.

Carto is available on Windows and Linux.

To launch Carto, you can extract the ZIP file that was made available to you and launch the executable file. Its important that the executable file stays in the same folder as the rest of the files.

You will also need to install the NDI binaries.

NDI Installation (Windows)

Follow the instructions available here

NDI Installation (Linux)

Follow the instructions available here

Hardware Requirements

Carto was mostly tested with NVIDIA RTX3060 graphics cards but less powerful GPUs can possibly work. The GPU must entirely support Vulkan 1.0.

Usage

UI Overview

Carto’s interface is made up of multiple sections.

Section 1 on the image is the 3D view. It contains all of the elements of the 3D cartography of the space : the devices, the point clouds and the crop region.

Section 3 is a toolbar with multiple menus and buttons.

Section 3 is the side bar which can be used to instanciate devices, adjust the crop region and declare network outputs.

3D View Controls

The 3D view contains different parts of you 3D cartography. What follows is a desctiption of the labeled elements of the screenshot.

1. Camera icons. These icons takes the color assigned to each device. Each camera icon points to the centroid of the point cloud captured by the device it represents.
2. Point cloud. Display of the point cloud captured by the camera. The point cloud takes the color assigned to its device.
3. Camera name. This name is displayed when the camera is selected or when the mouse cursor is hovering over the camera icon.
4. Camera manipulator gizmo. This gizmo is used to rotate and to translate a device and its point cloud. The arrows allows translations in each axis, the squares can be used to move the device in two axes at once and the arcs can be used to rotate the device.
5. Viewpoint camera gizmo. This gizmo can be used to manipulate the viewpoint camera. Click and drag on the circular surface of the gizmo to orbit the camera around the center of the display and click on the circles representing axes to obtain orthographic views.

7. Crop region. The gray volume represents the zone in which points will be kept. It is possible to get a gizmo on this zone by clicking on the handles

Toolbar

1. File menu. With all the classic option: save, save as and load.
2. Help button. Displays a menu that describes controls and shortcuts.
3. Configuration button: Displays the global configurations menu.
4. Log menu. Press this button to display a log widow where errors and wranings will be displayed.
5. Othographic views dropdown menu. The menu allows you to select either the perspective view or one of the orthographic views.

Menu

1. Camera tab. This tab contains the controls that are necessary to instanciate and configure new devices.
2. Crop region tab. This tab allows you to configure the size and position of the crop region.
3. Network output tab. This tab is used to instanciate and configures different network outputs.

Devices

Carto currently supports Orbbec Femto Megan and Hesai Pandar 40P. It is possible that other Orbbec devices and other Hesai devices can work but they have not been tested and may not be optimally supported.

Orbbec Femto Mega and generic device controls

Orbecc Femto Mega’s were the first devices to be integrated in Carto. When a device is created in Carto, the default device type is “Orbbec”, which refer to the Femto Mega.

Using an Orbbec is very simple: Once plugged to a PoE switch, the camera announces its identity to the network which allows Carto to auto-discover it.

Orbbec Femto Mega’s parameters in Carto

This annotated printscreen shows the controls specific to an Orbbec Femto Mega and other controls that are available for any devices.

1. Gizmo Button. Click on this button to obtain a gizmo on this device. It is possible to press Ctrl+Click to select multiple devices.
2. Camera Name This name appeas.
3. Device delete button. A deleted device will also be disconnected. It is possible to undo this operation with Ctrl+Z
4. Device activation toggle. A deactivated device will also be disconnected. It won’t use network bandwidth or computing resources.
5. Device frame rate display. This counter represents the number of point cloud frames that Carto receives from the device every seconds. Important: this value is independent of Carto’s internal framerate.
6. Device selection dropdown menu.
7. Refresh button. It is possible to force a device to reconnect by clicking on this button.
8. IPs dropdown. For Orbbec devices, this menu can be used to select an IP address from a list of detected devices.
9. Resolution dropdowns. Allows to select one of the valid resolutions for Orbbec Femto Megas.
10. Frame rate dropdown menu. Used to select a framerate for the Orbbec device. Automatically updates if the current resolution is not compatible with the current framerate.
11. Thinning menu. Available for every device. Removes points randomly from the displayed point cloud to thin it.

Bandwidth used by an Orbbec Femto Mega depending on its settings

This table can help you planify the network aspect of your installation.

Hesai Pandar 40P

Hesai 40P requires a bit more configuration.

After having plugged them in a network switch, find out its address and navigate to http://<your hesai's ip>/setting.html and change the destination ip and port (TODO: aller voir c’est quoi les champ et mettre un screenshot.)

This annotated screenshot shows the controls related to Hesai devices.

1. Hesai IP. It is not strictly necessary to put the right IP to receive points from a Hesai but having the right IP makes the “Open Web UI” button work properly.
2. Port. Port of the hesai. If you have multiple hesais on the same network, you need to change this port in order to have a unique port for each hesai.
3. Open Web UI. This button opens a web browser window that will show you the configuration of the Hesai that is at the displayed IP. You can further configure the hesai from that web interface.

Point Cloud Controls

These controls are available for all device types and allows you to control the points displayed in the 3D view.

1. RGB Color. RGBA color of the displayed points.
2. Calibrate Centroid. Click this button to retrigger a centroid computation. This computation is automatically done when Carto first receives points from a device but if the device was move or if the captured environment changed a lot, you may have to recalibrate the centroid.
3. Rotation Pivot. These buttons control the rotation pivot of the point cloud. When the rotation pivot is set to “Centroid”, the manipulator gizmo will appear at the “center” of the point cloud and the point cloud will rotate around this center. This is the default behaviour. When the rotation pivot is set to “Camera”, the gizmo appears on the camera icon and the point cloud will rotate as if you were rotating the device that is capturing it.
4. Translation fine tuning controls. Allows fine adjustments of the position of a point cloud
5. Rotation fine tuning controls. Allows fine adjustments of the rotation of a point cloud.
6. Reset buttons When a rotation or a translation is modified, these button allows you to reset them to 0. Note that any change to rotation and translation are also reversible with Ctrl-Z

Creation, configuration and calibration of a device

This small tutorial will guide you through the creation and calibration of two Orbbec Femto Mega cameras.

To calibrate devices, we advise you to start by deactivating the crop region. This allows you to freely move your point clouds in the 3D space without losing points due to the crop region.

To create a camera, go to the “Cameras” tab of the side bar and press on the “+” button. A menu item should be created. The Orbbec device type should be selected by default.

If Orbbec cameras are connected to your network, the IP dropdown should already contain their IP addresses. If you just plugged them in, wait a moment for them to start up and for Carto’s auto discovery process to find them. Selecting an IP should connect the camera to Carto and a point cloud should appear.

Start by trying to align the floor, walls or an easily identified object of your space. To help with the visualization, you can use the solo and mute buttons of each device. Orthographic views are very useful to align walls and floor.

For your second camera, try to align walls, floors or another object to the point cloud of your first camera.

You can then re-activate the crop region. Click on the handles of the crop region to get a gizmo that allows you to resize and move it.

The crop region allows you to crop out the floor, walls and other object to be able to get points only when someone is moving in the space.

Note that it is possible to select multiple objects with Ctrl+Click. This can help you to change the position of the cameras without losing your calibration.

Here is a video showing these steps:

Integration

This page contains tips to create a data capture system for a certain space using Carto.

Covering a space with Orbbec Femto Megas

The goal of a multi-camera system is to be able to capture visual data on the entire are where we envision people to interact with an installation. To make sure the system can provide reliable 3D data at all time, we need to optimise space coverage by positioning cameras in a careful manner.

According to the technical specifications of the Femto Mega devices, two modes of depth data capture are available: the Wide Field of View (WFoV) mode and the Narrow Field of View (NFoV) mode. Using the wide field of view widens the captured area near the device while the narrow filed of view can allow the device to get data from further distances. The technical specifications are displayed in the following table. Note that the distance recommended in the table is a conservative estimate: We have been able to get depth data from more than 8m of a device using the NFoV mode. Outdoors performances are very limited; it is preferable to use a lidar.

The disposition of the devices in the space will be determined by the size of the captation area and by the height of the ceilings. The higher the ceilings are, the easier it is to cover a large captation area with having to care about occlusions of the camera’s field of view. (This assumes we are able to mount cameras to the ceiling of a room).

The standard use case of a multi-camera system is to obtain visual information in a 3D zone of a height of at least 2 meters. This makes it possible to detect the bodies of every individual in the space. Low ceilings requires more cameras to offer a complete coverage of the area and to avoid occlusion.

Note that the number of cameras required and the required precision of the calibration depends on what kind of informations we want to get. The first objective will always be to get a spatial understanding of movement: Notably to be able to project the 3D location of a participant on an adjacent surface.

Troubleshooting

This document contains troubleshoothing advices for some issues you can encounter.

The received frame rate of a device is lower than its configured frame rate

Orbbec

If you are using Orbbec Femto Megas, make sure nothing else on the network is actively streaming data from the camera. These devices can stream to multiple softwares and computers but they are not able to do so while keeping an optimal send rate.

If no one else than you uses the network and the received rate is still to low, make sure you are not saturating your network’s bandwidth. At their maximum throughput (1024x1024 at 15fps), without color, the Orbbec Femto Mega can use 253 Mibps of bandwitdth. This means that a single Orbbec can saturate a 200 Mibps link and that adding another camera to the network will further degrade performances. Make sure to use a link of at least 1Gibps if you have a small amount of cameras. It is also possible to use many 1Gibps networks if the server on which Carto runs has multiple network interfaces. If you can’t use a network with more bandwidth, try reducing the resolution or the framerate of your cameras.

Hesai

If you are using a Hesai, the target framerate in Carto assumes that the lidar is configured at 1200 rpm. If the lidar is turning at 600 rpm you will have half the frame rate

My Orbbec cameras send at 0FPS after a while

We have notices that Orbbec cameras have a tendency to disconnect after a certain amount of time if they are not connected to a dedicated network. If your orbbec are connected to a busy network where other users or AV equipment generates traffic, try to create a dedicated network for them.