WebRTC Video

Provides low-latency video streaming over an end-to-end encrypted WebRTC connection. Once installed you can use the provided embedding instructions to embed the video widget for that robot in your own web application.

• Ultra low latency:
  • typical peer-to-peer latency:
    • local network connections: 30 ms
    • relay (TURN) connections: 60 ms (incl. 30 ms "ping time")
  • typical glass-to-glass, including ~100 ms USB camera latency: 130 ms
• Allows setting desired bit rate (in KB/s), e.g., for choosing between high definition and lower bandwidth cost
• Multi-camera support
  • Up to 10 cameras
  • Can be easily arranged into layouts using CSS
  • Configurable relative priorities for bandwidth allocation per camera
• Supports various video sources:
  • Video4Linux cameras, i.e., most USB cameras
    • allows you to select from list of resolutions and frame rates supported by your cameras
  • ROS and ROS2 image topics, incl. bayer encoded ones, and pre-encoded h264 streams
  • RTSP streams, e.g., from IP cameras
  • custom GStreamer source pipelines
    • including zedsrc for ZED cameras (requires adding libgomp.so.1 to ld.so.preload)
• Efficient and responsive video encoding
  • Uses h264 for efficient compression
  • Hardware acceleration on Nvidia platforms (e.g., Jetsons), Rockchip based platforms (e.g., Orange Pi, Firefly), and Intel VA-API compatible chipsets (e.g., HD Graphics)
  • Minimizes bandwidth use when video scene is static
  • Congestion control: automatically adjust bit rate to account for network conditions
• Robust against even heavy packet loss
• Automatically reconnects after network loss
• Encrypted end-to-end
  • To learn more about the security model of WebRTC and why it is safe see, e.g., here.
• No bandwidth cost when sender and receiver are on the same network
• Audio streaming (beta)
• Supervisor UI:
  • A UI component listing all ongoing sessions on a robot
  • Allows joining an ongoing session without increasing bandwidth or CPU usage on the robot

When running in a self-hosted deployment offline, i.e., not connected to the Internet, the two WebRTC peers, robot + web user, can now discover each other via a new, locally hosted STUN server included in the cloud capability. This is used automatically when the robot is unable to reach the default TURN server hosted by us. This means you can now use Remote Teleop in local networks that are not connected to the Internet. Note, however, that your server still needs to go online every few days to receive an updated billing token for your robots' capabilities.

Requires gstreamer 1.16 (Ubuntu 20.04) or later.

During the installation process the Transitive agent will try to install all required dependencies into its sandbox environment. If this fails or if your build and deployment process makes it preferable you can pre-install them manually:

sudo apt install build-essential pkg-config fontconfig git gobject-introspection gstreamer1.0-x gstreamer1.0-libav gstreamer1.0-nice gstreamer1.0-plugins-bad gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-tools libgstreamer1.0-0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev libgirepository1.0-dev libc-dev libcairo2 libcairo2-dev

If you are running Transitive inside a Docker container and want to use USB cameras, then be sure to add the following to your docker run command (or similarly for docker-compose):

-v /run/udev:/run/udev # required by gst-device-monitor-1.0 to enumerate available devices and supported resolutions

See Video Configuration.

Each <video> element generated by the front-end web component will have a unique class name webrtc-videoN, where N enumerates the elements starting at 0. This makes it easy to arrange and style these elements using CSS. For instance, these CSS rules would create a layout where one camera, e.g., the front camera, is large on top, and at the bottom we have left, back, and right-viewing cameras.

webrtc-video-device video { position: absolute }
webrtc-video-device .webrtc-video0 { width: 960px; height: 720px; }
webrtc-video-device .webrtc-video1 { top: 720px; }
webrtc-video-device .webrtc-video2 { top: 720px; left: 640px; }
webrtc-video-device .webrtc-video3 { top: 720px; left: 320px; width: 320px; }

In addition, the div element immediately wrapping these video elements has the class name webrtc-video-wrapper. This makes it possible to use apply various CSS layout features such as flexbox or grid layouts, e.g., the following would create a layout where the front facing camera is large in the middle, left and right cameras are to the sides on top, and the backward facing camera is in the bottom left. The bottom right is left black here but would make for a good place to show a map component.

.webrtc-video-wrapper {
  display: grid;
  grid-gap: 10px;
  grid-template-columns: 1fr 1fr 1fr 1fr;
  grid-template-rows: 1fr 1fr;
  grid-template-areas:
  "left front front right"
  "back front front .";
}
.webrtc-video0 { grid-area: front; }
.webrtc-video1 { grid-area: left; }
.webrtc-video2 { grid-area: right; }
.webrtc-video3 { grid-area: back; }
video {
  width: 100%;
  height: 100%;
  object-fit: cover;
}

In React, this capability can be embedded using the TransitiveCapability tag from the @transitive-sdk/utils-web package. When doing so, you can provide an additional object as props containing callbacks you would like to receive, e.g.:

<TransitiveCapability jwt={jwt} timeout="1800"
  handlers={{
    // receive connection state events:
    onConnectionStateChange: (state) => console.log({state}),
    // continuously receive lag information:
    onLag: (lag) => console.log({lag}),
    // continuously receive stats from the webrtc connection:
    onStats: (stats) => console.log({stats}),
    // receive tracks as they are created
    onTrack: (track, mid) => console.log({track, mid}),
    // be notified when the user clicks on the video
    onClick: (clickEvent) => console.log({clickEvent}),
  }} />

The supervisor UI component allows you to join an ongoing session. It does so by forwarding the streams being watched from the watchers computer directly, i.e., it does not increase resource or bandwidth usage on the robot itself.

Like all UI components, the Supervisor UI can be embedded on other pages as well. When doing so, you can set the following options to automatically join the latest ongoing session on the device, and, optionally, to select only a subset of all stream (when the session has more than one stream/camera):

• auto: set this if you want to auto-join the latest session
• streams: a comma separated list of stream indices, e.g., "0,2"

This allows you, for instance, to create a page where you are showing one stream for each ongoing session. That page will auto-update as sessions start and stop, giving you a bird-eye overview of what is happening with your fleet.

• For ROS2, uses newer version of rclnodejs with pre-built binaries on Humble+

• v1.3.3

  • Fixes a bug in lag-detection when receiving audio

• v1.3.4

  • Improves peer-to-peer connection negotiation on unreliable networks
  • Dark mode for session history

• v1.3.5

  • Fixes an issue on older versions of NVIDIA Jetpack when using compressed ROS topics

• v1.3.11

  • Adds new prop sessionname that let's you assign a name to your session. You can use it in the Supervisor UI to indicate which session to join.

• Adds support for pipewire video devices (default in Ubuntu 24.04+).

• Adds partial support for UVC H264 devices (Ubuntu 24+).

• Now shows display names for video devices in UI.

• v1.2.2

  • Fixes a bug in the packet loss mitigation logic of Ubuntu 22+.

• v1.2.3

  • Fixes a bug that prevented using more than six streams.
  • Fixes a bug in the auto-bitrate logic that resulted in a rate that was too low when using multiple streams.

• Uses hardware acceleration on Nvidia platforms for decoding JPEG compressed image ROS topics

• v1.1.4

  • Allow setting framerate for ROS topic sources e.g., framerate="30/1".
  • Show per-track FPS in status badge tooltip.

• v1.1.5

  • Adds new per-stream option priority that allows you to set the relative priority of that stream for allocating available bandwidth (default = 1).

• No changes.

• v1.0.2

  • Adds support for ROS 2 Jazzy.

• v1.0.3

  • Adds new parameters blacklist_netmask and whitelist_netmask that let you black list (resp. white list) robot-side netmasks to use for establishing the WebRTC peer-to-peer connection. This can be useful, e.g., to prevent the use of VPN interfaces (e.g., blacklist_netmask: '10.0.0.0/8,172.0.0.0/8').

• Adds new "local mode".
• Fixes a bug in dynamic bitrate control when using audio.

• Added support for JPEG compressed ROS image topics (sensor_msgs/CompressedImage).

• Added hardware acceleration for Intel devices supporting VA-API.

• v0.23.1:

  • Fixes an issue on some platforms with VA-API hardware acceleration.

• v0.23.2:

  • Adds the ability to skip source verification for ROS topics. This is required by some hardware encoders, incl. some VA-API ones, that do not handle the switch between the fall-back message ("Waiting for ROS topic..") and the actual stream correctly.

• v0.23.3:

  • Small performance improvement for ROS topics.

• Added bandwidth optimization.
  • This saves bandwidth when the scene in the video is mostly static by using variable bitrate encoding. When using the "high" setting for this, it can reduce bandwidth to as low as 5KB/s.
• Reduced latency to around 100ms.
• Increased robustness:
  • Better error handling and reporting when sessions fail.
  • Eliminated more cases where a request could get "stuck", blocking future sessions using the same parameters.
  • Reduced delay in starting the stream in the face of packet loss when using NVIDIA hardware encoders.

• v0.22.1:

  • add quantizer setting (bandwidth optimization) to config provided in Embed
  • correctly set bitrate, peak-bitrate, and/or maxbitrate on NVIDIA jetson devices

• v0.22.2:

  • ease-of-use: update video sources on page load and right away when a session ends, to update busy status of v4l2 devices.
  • new feature: auto-bitrate mode. This allows you to leave the bitrate unspecified, letting the capability decide automatically what an appropriate bitrate is given the video sources you selected. This, together with the new bandwidth optimization feature, makes it extremely easy to get good quality streams without any tuning.

• v0.22.3:

  • Ease-of-use: Automatically recover from busy USB cameras and timeouts receiving ROS messages
  • Namespace ROS 2 nodes in /transitive

• v0.22.4:

  • Fixes a race condition with ROS topics on NVIDIA hardware encoders (VIC only, dGPU was not affected).

• v0.22.5:

  • Fixes a bug when doing local recording of ROS topic sources
  • Sets the default MTU to 1,200, adds new parameter rtpMtu to overwrite this

• v0.22.6:

  • Fixes a bug when using rostopic-h264 sources, i.e., ROS topics that contain pre-encoded h264 streams

• Improves CPU efficiency of h264 encoding by 3x when using software encoder
• Adds ability to customize test-image resolution and framerate

• Support for ROS topics carrying pre-encoded h264 streams

• v0.20.1:

  • Avoids high memory usage when establishing the connection takes long (e.g., due to packet loss) when using ROS topics

• Supervisor UI, embedding:
  • auto-join last active session
  • only subscribe to subset of session streams
• Made ready for upgrade to Node 20

• Replace video-streams with an error message if they fail to start within a timeout (currently 2 seconds). This is particularly useful when you stream multiple cameras but one fails to start, e.g., due to USB issues.

• Improved congestion control: accounting for static network delay
• New debug option: disable congestion control
• Increased default bitrate to 100KB/s

• New feature: support for streaming audio as well
• New feature: supervisor UI

• v0.16.4:

  • Fixes a problem with h264 streams via v4l2loopback

• v0.16.8:

  • Fixes reactivity, e.g., when switching devices (in JWT) or input sources

• v0.16.10:

  • Fixes an issue regarding the installation of ROS 2 bindings

• v0.16.11:

  • Time out if ROS topic is not published, but still show other streams

• New feature: ability to record locally to disk on robot/device
• New feature: ability to specify custom source and sink pipelines
• New feature: support for Bayer encoded ROS image topics

• v0.15.1:

  • Automatically recover from interruptions during updates

• New feature: hardware acceleration on Rockchip based boards
• Fixed a bug preventing the connection to establish from mobile browsers
• Fixed a bug preventing the embedding of video streams using ROS 2 topics as video source