Point-cloud segmentation, or “cloudseg” for short, was one of the more challenging projects at Luxolis. The idea was relatively simple though.

You are given a colored 3D mesh reconstruction of an indoor space (e.g. a living room). You are to assign each vertex the following labels: class label, instance label, and part label.

Let’s get to the example of a living room. The first stage - class label - assigns each vertex (a point in space), one of the predetermined classes (wall, floor, ceiling, chair, etc.). The second stage is actually determining which two points sharing the same class, actually are two different objects.

Various stages of the Mask3D segmentation. Notice that on the right-hand panel the two instances of the same class ("sofa") are separated.

Mask3D segmentation is much more robust than the current state-of-the-art (Softgroup)

A small, investor-focused showcase of Mask3D segmentation (a small living room)

The core of the model - Sparse convnet (Minkowski engine) + Decoder transformer.

A Live demo

Follow the instructions down below in order to generate 3D point cloud segmentation, works both indoors and outdoors. Make sure to use the supported 3D point cloud formats, and to orient the cloud correctly (z-axis points towards positive vertical direction).