Driver for the Intel realsense cameras.

The component is able to stream the color, infrared, 3D images as well as gyroscopes and accelerometer sensors of one or multiple realsense devices. For streaming color or infrared images, see stream_video (function). For 3D images, see stream_depth (function). For gyroscopes and accelerometers, see stream_motion (function).

Multiple image (multiple out) output ports can be published, organized into individual banks. Each bank contains a sequence of coherent images, e.g. time synchronized.

The internal architecture implements a graph of named source and sink slots, which can be published individually into images of a bank. Hardware-dependent functions can stream raw data into slots, which can then filtered or otherwise processed by generic services into other slots and eventually published with publish (activity).

Note that the term slot in this documentation always refers to an internal image of the processing graph, while the term bank refers to a published image in a bank of the port image (multiple out).

In addition to hardware-specific services, there are generic services, common to all camera interface providers. These include basic filtering, recording and publishing functions.

Ports

image (multiple out)

Data structure

  • struct ::or_image::bank image

    • struct ::or::time::ts ts

      • long sec

      • long nsec

    • sequence< struct ::or_image::frame, 8 > images

      • struct ::or::time::ts ts

        • long sec

        • long nsec

      • struct ::or_image::info info

        • unsigned long seq

        • optional< struct ::or::sensor::pixmap_roi > roi

          • unsigned long subsx

          • unsigned long subsy

          • unsigned long x

          • unsigned long y

          • unsigned long width

          • unsigned long height

        • optional< struct ::or_image::distortion > distortion

          • enum ::or_image::distortion_model model ∈ { R3T2 }

          • sequence< double, 8 > coeffs

        • optional< struct ::or_image::intrinsic > intrinsics

          • double fx

          • double fy

          • double cx

          • double cy

        • optional< struct ::or_image::extrinsic > extrinsics

          • struct ::or::t3d::pos T

            • double x

            • double y

            • double z

          • struct ::or::t3d::att R

            • double qw

            • double qx

            • double qy

            • double qz

      • struct ::or::sensor::pixmap pixmap

        • unsigned long seq

        • unsigned long width

        • unsigned long height

        • unsigned long stride

        • enum ::or::sensor::pixmap::encoding pixfmt ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }

        • sequence< octet > data

Published image banks.

Images in a bank are considered as a coherent sequence of individual frames, hence sharing a common ts timestamp information.

Each image in the array contains its own metadata information info and a pixmap. The seq member represents a sequence number and different images read from the same bank index with the same seq number should be considered identical. This is especially useful to check if the calibration data has changed.

info.roi indicates that the image data was extracted from a bigger image. In particular, the intrinsics refer to the original image dimensions.

info.distortion represents the image distortion information and the associated correction model.

info.intrinsics is the intrinsic matrix mapping from 2D coordinates to 3D information, according to the following model where u,v are image pixel coordinates, w a free parameter and X, Y, Z 3D coordinates in the camera frame:

| w.u   | fx  0 cx | | X
| w.v = | 0  fy cy |.| Y
| w     | 0   0  1 | | Z

info.extrinsics is a 3D transformation converting 3D coordinates from the camera frame to another frame common to all images in a bank:

| X'     | X
| Y' = R.| Y + T
| Z'     | Z

pixmap.data contain the actual pixel values, arranged as a row-major 2D array of height×width pixels. The width in bytes of each pixel row is indicated by stride.

Each pixel has the format pixmap.pixfmt, defining the number of channels: MONO is 1 integer grayscale channel, RGB or BGR are 3-color integer channels, possibly with an additional transparency channel A. Z and XYZ are 1 and 3 floating point depth channels repectively. The number of bits encoding each channel value is suffixed. JPEG indicates a compressed RGB color image.

accel (out)

Data structure

  • struct ::or_pose_estimator::state accel

    • struct ::or::time::ts ts

      • long sec

      • long nsec

    • boolean intrinsic

    • optional< struct ::or::t3d::pos > pos

      • double x

      • double y

      • double z

    • optional< struct ::or::t3d::att > att

      • double qw

      • double qx

      • double qy

      • double qz

    • optional< struct ::or::t3d::vel > vel

      • double vx

      • double vy

      • double vz

    • optional< struct ::or::t3d::avel > avel

      • double wx

      • double wy

      • double wz

    • optional< struct ::or::t3d::acc > acc

      • double ax

      • double ay

      • double az

    • optional< struct ::or::t3d::aacc > aacc

      • double awx

      • double awy

      • double awz

    • optional< struct ::or::t3d::pos_cov > pos_cov

      • double cov[6]

    • optional< struct ::or::t3d::att_cov > att_cov

      • double cov[10]

    • optional< struct ::or::t3d::att_pos_cov > att_pos_cov

      • double cov[12]

    • optional< struct ::or::t3d::vel_cov > vel_cov

      • double cov[6]

    • optional< struct ::or::t3d::avel_cov > avel_cov

      • double cov[6]

    • optional< struct ::or::t3d::acc_cov > acc_cov

      • double cov[6]

    • optional< struct ::or::t3d::aacc_cov > aacc_cov

      • double cov[6]

gyro (out)

Data structure

  • struct ::or_pose_estimator::state gyro

    • struct ::or::time::ts ts

      • long sec

      • long nsec

    • boolean intrinsic

    • optional< struct ::or::t3d::pos > pos

      • double x

      • double y

      • double z

    • optional< struct ::or::t3d::att > att

      • double qw

      • double qx

      • double qy

      • double qz

    • optional< struct ::or::t3d::vel > vel

      • double vx

      • double vy

      • double vz

    • optional< struct ::or::t3d::avel > avel

      • double wx

      • double wy

      • double wz

    • optional< struct ::or::t3d::acc > acc

      • double ax

      • double ay

      • double az

    • optional< struct ::or::t3d::aacc > aacc

      • double awx

      • double awy

      • double awz

    • optional< struct ::or::t3d::pos_cov > pos_cov

      • double cov[6]

    • optional< struct ::or::t3d::att_cov > att_cov

      • double cov[10]

    • optional< struct ::or::t3d::att_pos_cov > att_pos_cov

      • double cov[12]

    • optional< struct ::or::t3d::vel_cov > vel_cov

      • double cov[6]

    • optional< struct ::or::t3d::avel_cov > avel_cov

      • double cov[6]

    • optional< struct ::or::t3d::acc_cov > acc_cov

      • double cov[6]

    • optional< struct ::or::t3d::aacc_cov > aacc_cov

      • double cov[6]

Services

publish (activity)

Inputs

  • string<16> slot (default "") Slot name

  • string<16> bank (default "") Published bank name

  • octet index (default "0") Published bank index
Throws

  • exception ::camera::einval

    • string<128> what
Context

Start publishing a named slot to an output port.

This exports the internal image present in the named slot into the output port bank, at the position index within the sequence of images of the port (see image (multiple out) structure).

slot must be the name of an output slot of a hardware streaming function or a software processing function. bank name can be chosen freely and is reflected in the output port name. index can also be chosen freely depending on the desired configuration of the output port.

If multiple images have to be published in a time-synchronized manner, they should be placed in the same bank, at different index position. For instance, if to slots left and right are streamed by a stereoscopic hardware, they can be placed in the same port named stereo by using two publish requests with slot left, bank stereo, index 0 and then slot right, bank stereo, index 1.

If multiple images have to be published but do not require specific time syncronization, they can be published to different ports. This can make the client access more efficient, as each client can read only the images in which it is interested in. For instance, to publish the above example images left and right to port A and B, use two publish requests with slot left, bank A, index 0 and then slot right, bank B, index 0.

unpublish (activity)

Inputs

  • string<16> bank (default "") Published bank name

  • octet index (default "0") Published bank index
Throws

  • exception ::camera::einval

    • string<128> what
Context

Stop publishing a port.

See publish (activity).

jpeg (function)

Inputs

  • string<16> input (default "") Input slot name

  • unsigned short quality (default "80") Compression quality [0-100]

  • string<16> output (default "") Output slot name
Throws

  • exception ::camera::einval

    • string<128> what

Compress a slot using JPEG compression.

This converts the input slot to the output slot output using JPEG compression, with the specified quality.

record_mpeg (function)

Inputs

  • string<16> input (default "") Input slot name

  • unsigned short width (default "640") Image width

  • unsigned short height (default "480") Image height

  • unsigned short fps (default "25") Image height

  • string<128> output (default "") Output file (.avi extension)
Throws

  • exception ::camera::einval

    • string<128> what

Record a slot using MPEG compression in an AVI container.

The input slot is first resized to the desired width and height and compressed into the output file. The playback video frame rate is fixed and equal to fps, no matter the input slot rate. The output file name must end with .avi.

Additionnaly, an output slot named after the output file name, with any leading slash (/) stripped, contains the last compressed image, appropriately scaled to width and height.

cancel (function)

Inputs

  • string<16> slot
Throws

  • exception ::camera::einval

    • string<128> what

  • exception ::realsense::e_rs

    • string<128> what

Cancel all jobs publishing to slot.

The effect of this function is that slot will no longer be updated and the associated processing will be cancelled. This is useful to reconfigure the processing graph with e.g. different parameters or different inputs.

status (function)

Inputs

  • string<16> slot
Throws

  • exception ::camera::einval

    • string<128> what

Raise any processing errors associated with slot.

If, for some reason, a slot fails to be updated, the error will be available by checking the result of this function. Processing errors typically include wrong pixel format for a filter or wrong image dimensions.

get_imu_noise (attribute)

Outputs

  • double gstddev[3] Gyroscope error model

  • double astddev[3] Accelerometer error model

set_imu_noise (attribute)

Inputs

  • double gstddev[3] Gyroscope error model

    • double [0] (default "0.1")

    • double [1] (default "0.1")

    • double [2] (default "0.1")

  • double astddev[3] Accelerometer error model

    • double [0] (default "0.1")

    • double [1] (default "0.1")

    • double [2] (default "0.1")

get_streams (function)

Outputs

  • sequence< enum ::realsense::stream, 16 > streams ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

Get available streams.

get_stream_indices (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< long, 16 > indices

Get available indices for a given stream type.

get_stream_fps (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< long, 16 > fps

Get available streaming rates for a given stream type.

get_video_stream_formats (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< enum ::or::sensor::pixmap::encoding, 16 > format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }
Throws

  • exception ::realsense::einval

Get available video streaming formats for a given stream type.

get_depth_stream_formats (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< enum ::or::sensor::pixmap::encoding, 16 > format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }
Throws

  • exception ::realsense::einval

Get available video streaming formats for a given stream type.

get_video_stream_sizes (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< struct ::realsense::video_size, 16 > sizes

    • long width

    • long height
Throws

  • exception ::realsense::einval

Get available video streaming sizes for a given stream type.

get_depth_stream_sizes (function)

Inputs

  • enum ::realsense::stream type Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }
Outputs

  • sequence< struct ::realsense::video_size, 16 > sizes

    • long width

    • long height
Throws

  • exception ::realsense::einval

Get available depth streaming sizes for a given stream type.

stream_video (function)

Inputs

  • enum ::realsense::stream type (default "::realsense::STREAM_COLOR") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

  • short index (default "-1") Index for streams of the same type (-1 for any)

  • short width (default "0") Image width (0 for any)

  • short height (default "0") Image height (0 for any)

  • enum ::or::sensor::pixmap::encoding format (default "::or::sensor::pixmap::ANY") Pixel format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }

  • short fps (default "0") Frames per second (0 for any)

  • string<16> name (default "color") Slot name
Throws

  • exception ::realsense::einval

  • exception ::realsense::e_rs

    • string<128> what

Stream video data.

The stream type should be one of STREAM_COLOR or STREAM_INFRARED and index one of the available index returned by get_stream_indices (function), or -1 for a default one. Image is made available in the named output slot. The name of the slot can be chosen freely.

If the device has multiple streamable images, stream_video (function) can be invoked multiple times to stream them all. In this case, you need to use the index parameter explicitly to specify which image to stream to which slot. For instance, invoking stream_video (function) twice with the parameters type STREAM_INFRARED, index 1 and slot left and then with type STREAM_INFRARED, index 2 and slot right will stream the two infrared images of a D435 camera to the internal slots left and right.

Each of the slots created by this request is available for further processing (e.g. with jpeg (function), record_mpeg (function), …​) or publishing with publish (activity).

stream_depth (function)

Inputs

  • enum ::realsense::stream type (default "::realsense::STREAM_DEPTH") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

  • short index (default "-1") Index for streams of the same type (-1 for any)

  • short width (default "0") Image width (0 for any)

  • short height (default "0") Image height (0 for any)

  • enum ::or::sensor::pixmap::encoding format (default "::or::sensor::pixmap::ANY") Pixel format ∈ { ANY, MONO8, RGB8, BGR8, RGBA8, BGRA8, MONO16, Z32, XYZ32, JPEG, INVALID }

  • short fps (default "0") Frames per second (0 for any)

  • string<16> name (default "color") Slot name
Throws

  • exception ::realsense::einval

  • exception ::realsense::e_rs

    • string<128> what

Stream depth data.

The stream type should be STREAM_DEPTH and index one of the available index returned by get_stream_indices (function), or -1 for a default one. The stream format must be one of Z32 or XYZ32 (or ANY which will select Z32). Image is made available in the named output slot.

See also stream_video (function) for further explanations about streaming multiple depth images, processing or publishing them.

Depth information for Z32 format and 3D coordinates for XYZ32 format are expressed in meters (m).

stream_motion (function)

Inputs

  • enum ::realsense::stream type (default "::realsense::STREAM_GYRO") Stream type ∈ { STREAM_ANY, STREAM_COLOR, STREAM_INFRARED, STREAM_DEPTH, STREAM_GYRO, STREAM_ACCEL, STREAM_INVALID }

  • short index (default "-1") Index for streams of the same type (-1 for any)

  • short fps (default "0") Frames per second (0 for any)
Throws

  • exception ::realsense::einval

  • exception ::realsense::e_rs

    • string<128> what

Stream motion data.

The stream type should be STREAM_GYRO or STREAM_ACCEL and index one of the available index returned by get_stream_indices (function), or -1 for a default one. Data is published the corresponding port: gyro (out) or accel (out).

Tasks

worker0

Context

Worker task 0 processing and publishing frames.

Worker tasks are independant of each other can do parallel processing of multiple images.

worker1

Context

Worker task 1 processing and publishing frames.

main

Context