C++ Neural Networks Tutorial

Load and run a neural network

The input and output data of neural networks are called tensors, which are essentially N-dimensional arrays. FAST will automatically convert input Image objects to Tensor data objects and feed that to the neural network.

// Set up image importer to load an ultrasound image
auto importer = ImageFileImporter::create(Config::getTestDataPath() + "/US/JugularVein/US-2D_100.mhd");

// Load neural network from a file and connect it to the importer
auto network = NeuralNetwork::create(Config::getTestDataPath() + "NeuralNetworkModels/jugular_segmentation.onnx")
    ->connect(importer);

// Run the pipeline and get the resulting tensor
auto tensor = network->runAndGetOutputData<Tensor>();
std::cout << "Shape of output tensor is " << tensor->getShape()->toString() << std::endl;

Inference engines

FAST includes three different inference engines, Google's TensorFlow, Intel's OpenVINO and NVIDIA's TensorRT. GPU processing with TensorFlow and TensorRT requires CUDA and cuDNN.

Depending on the file format of the neural network you load, FAST will select the "best" inference engine which supports that format. Currently the following formats are supported:

• ONNX -> OpenVINO, TensorRT
• Protobuf (.pb) -> TensorFlow
• SavedModel -> TensorFlow
• OpenVINO Intermediate Representation (IR) -> OpenVINO

You can also manually specify which inference engine you want to use:

network->load(
    Config::getTestDataPath() + "NeuralNetworkModels/jugular_segmentation.onnx",
    {}, {}, "OpenVINO"
);

Image segmentation

The output data of the NeuralNetwork process object is tensors. In the case of image segmentation, the shape of the output tensor is often H x W x C, where H and W are the height and width of the output segmentation, and C is the number of segmentation classes. It is possible to visualize this raw segmentation tensor directly as a heatmap using the HeatmapRenderer, but usually you want to convert this tensor to a Segmentation image. When converting a segmentation tensor to a segmentation image, it is common to select the class with maximum confidence with or without a threshold. In FAST you can convert from a segmentation tensor to a Segmentation image data object, by using the TensorToSegmentation process object, then you can visualize the Segmentation using the SegmentationRenderer, here is a complete example of this:

// Set up image importer to load an ultrasound image
auto importer = ImageFileImporter::create(Config::getTestDataPath() + "/US/JugularVein/US-2D_100.mhd");

// Load neural network from a file
auto network = NeuralNetwork::create(Config::getTestDataPath() + "NeuralNetworkModels/jugular_segmentation.onnx")
    ->connect(importer);
network->setScaleFactor(1.0f/255.0f); // Add a preprocessing step: multiply each pixel with 1/255, thus normalizing the intensity

// Convert tensor to segmentation
auto tensor2seg = TensorToSegmentation::create()
    ->connect(network);

// Setup visualization
auto renderer = ImageRenderer::create()
    ->connect(importer);

auto segRenderer = SegmentationRenderer::create()
    ->connect(tensor2seg);
segRenderer->setColor(1, Color::Red()); // Set color for class 1
segRenderer->setColor(2, Color::Blue()); // Set color for class 2

auto window = SimpleWindow2D::create()
    ->connect({renderer, segRenderer});
window->run();

For convenience you can also use the SegmentationNetwork process object, which extends NeuralNetwork by applying TensorToSegmentation on the output as shown here:

// Set up image importer to load an ultrasound image
auto importer = ImageFileImporter::create(Config::getTestDataPath() + "/US/JugularVein/US-2D_100.mhd");

// Use SegmentationNetwork instead of NeuralNetwork which applies TensorToSegmentation
auto network = SegmentationNetwork::create(Config::getTestDataPath() + "NeuralNetworkModels/jugular_segmentation.onnx")
    ->connect(importer);
network->setScaleFactor(1.0f/255.0f); // Add a preprocessing step: multiply each pixel with 1/255, thus normalizing the intensity

// Setup visualization
auto renderer = ImageRenderer::create()->connect(importer);

auto segRenderer = SegmentationRenderer::create()->connect(network);
segRenderer->setColors({{1, Color::Red()}, {2, Color::Blue()}});

auto window = SimpleWindow2D::create()
    ->connect(renderer)
    ->connect(segRenderer);
window->run();

Image classification

For image classification you may use the ImageClassificationNetwork convenience class which converts the output tensor to a list of class names and confidence values.

Optical flow

For optical flow motion estimation you may use the FlowNetwork convenience class which will convert the output tensor to a 2 channel Image object. Optical flow networks typically needs 2 consecutive image frames as input, this can be achieved using the ImagesToSequence process object. You can then visualize the flow image using the VectorFieldRenderer or the VectorFieldColorRenderer. Here is an example showing how a streaming pipeline with an optical flow network can look:

// Setup image stream
auto streamer = ImageFileStreamer::create("path/to/image/sequence/frame_#.mhd");

// Convert image stream to a stream of sequences with length 2
auto sequence = ImagesToSequence::create(2)
    ->connect(streamer);

// Setup neural network
auto flowNetwork = FlowNetwork::create("some-flow-network.onnx")
    ->connect(sequence);

// Setup visualization
auto imageRenderer = ImageRenderer::create()->connect(streamer);

auto flowRenderer = VectorFieldColorRenderer::create()->connect(flowNetwork);

auto window = SimpleWindow2D::create()
    ->connect({imageRenderer, flowRenderer});
window->run();

Bounding box detection

For bounding box detection you may use the BoundingBoxNetwork convenience class which converts the output tensor to a BoundingBoxSet data object. The BoundingBoxSet can be visualized using the BoundingBoxRenderer.

Multi-input and multi-output networks

When you load a neural network from file in FAST, it will create an input port and output port for every input and output node in the neural network. This enables you to create complex processing pipelines with multi-input and multi-output neural networks.

For example, if you have a NeuralNetwork which takes an image and a tensor as inputs, and provides two output tensors, it may look something like this:

auto importer = ImageFileImporter::create("someimage.jpg");

auto tensor = Tensor::create({1.0f, 0.0f, 3.0f}, TensorShape({1, 3}));

auto network = NeuralNetwork::create("some-multi-input-output-network.onnx")
    ->connect(0, importer)
    ->connect(1, tensor);

network->run(); // Run neural network

// Get output data
auto tensor1 = network->getOutputData<Tensor>(0);
auto tensor2 = network->getOutputData<Tensor>(1);

Batch processing

With batch processing of neural networks, you can process several samples at a time. This may give a speedup with a GPU, as the samples may processed in parallel and thus utilize the processor more efficiently. To do batch neural network processing in FAST use the Batch data object, and add your Image or Tensor data objects to it. Then give this Batch data object to the NeuralNetwork instead.

// Assuming we have 3 image data objects stored in variables image1, image2, image3:
// Create batch data object, and add the 3 images
auto batch = Batch::create({image1, image2, image3});

// Give the batch to your neural network
auto neuralNetwork = NeuralNetwork::create("some-neural-network.onnx")
    ->connect(batch);

// The output will be a Batch object as well, consisting of one output tensor for each input sample
auto outputBatch = neuralNetwork->runAndGetOutputData<Batch>();

Sequence processing

If you have a neural network which process a sequence of images or tensors, you may, in similar way as with batch processing, add your Image or Tensor data objects to a Sequence data object.

To create a pipeline which converts an image stream to a stream of sequences with a specific length you can use the ImagesToSequence process object as shown in the optical flow example above.

Custom operators and plugins

If your model needs a custom operator, layer or plugin, you can specify these in the NeuralNetwork::create function when you load your model. Specify the full path to your plugin/operator files (.so/.dll/.xml files), like so:

auto network = NeuralNetwork::create("some-network-model.pb", "", {}, {}, {"path/to/plugin1.so", "path/to/plugin2.so"});

Access tensor data

Next steps

• See more C++ Tutorials.
• Check out some C++ Examples.
• Review Concepts & Glossary used in FAST.