Possible improvements for dense matching

[paulinus]

The current dense matching implementation uses the PatchMatch variant from Shen13

Here is a list of possible improvements to the current implementation.

• Init scores scores properly. Right now scores are initialized to 0. They should be initialized by computing the score of the initial depth map.
• Init depthmaps from sparse point cloud. Depthmap is currently initialized using random values to explore the search space. An alternative would be to initialize it using the SfM points. For example, interpolating the depth of those points for all pixels.
• Ignore homogeneous pixels. Pixel on the sky or on white walls have very little texture information and usually produce no match or an erroneous one. It would be better to not use them at all. Should try removing patches with constant color or constant gradient.
• Sample views instead of checking them all. Currently, if more than one view are used as a reference, the score of a given depth plane is best correlation score amount the reference views. All views are checked in order to compute the best correlation. A faster alternative would be to sample the reference view randomly in a similar way we do for depth and normal. After each all random samples of depth and normal are done, we can sample a random view and check if matching score is better on that other view.
• Filter out depth values without supporters on their neighborhood. When cleaning a depthmap, additionally to checking that other depthmaps agree with a depth value, check that the neighborhood of the point on its depthmap and on the others has enough points/depths nearby. That should remove many more outliers.
• Average matching points instead of keeping just one of them when merging.
• Striding depth computation. Since we use a 7x7 correlation kernel, very little change between one pixel and the next one. One could try computing the depth only every each other pixel to save time and then interpolating. Note that this is different than reducing the image size.
• Approximate the plane induced homography by an affinity. We only use the homographies locally around one pixel so an affinity would be a good approximation. Given the affinity, the warpping for every pixel can be computed by adding vectors.

[xialang2012]

How can I run the dense module? Could you give some directions?

[paulinus]

you can run

bin/run_all path_to_data
bin/opensfm undistort path_to_data
bin/opensfm compute_depthmaps path_to_data

And you should get the resulting point cloud depthmaps/merged.ply

[xialang2012]

got it, thanks.