Surging Forward

Unhappy with the current state of RTP / RTSP video streaming libraries for mobile applications, we decided to set out on our own. The result is Surge and we think it's awesome.

Introducing Surge

While developing the cross-platform victorGo mobile application for a client, we were somewhat perplexed at the lack of tools out there for video streaming over RTSP on mobile. Sure there are libraries available such as Live555 but none with a focus on performance in resource-constrained environments such as mobile and often with licensing structures that don't easily facilitate a public release via the Apple App Store or Google Play Store. We knew that we could do better than this, so we set out to create Surge.

From the outset, our goal with Surge was to create an RTP / RTSP streaming solution with a keen focus on performance and ease of use, while remaining easily extensible should a more specific application be required. Part of the vision was to create a solid backbone from which the creation of platform-specific SDKs would be simple and natural. With its core firmly rooted in C++, we now have APIs for iOS and macOS via Objective-C and Swift, Android via Java and Xamarin via C# but we plan to expand further into areas such as tvOS and Chromecast. Furthermore the popular H.264, MPEG-4 and MJPEG formats are currently supported, but Surge's flexible and extensible nature makes it simple to supply additional decoders to support more formats in the future.

Full details about Surge's various features can be found on our website. There you will also find case studies for victor Go and Blueye robotics, showcasing the power of Surge and its simplicity in integration, even into existing large-scale projects, where it can reduce codebase and complexity by abstracting much of the video streaming nitty-gritty allowing developers to focus on delivering the core aspects of their vision. CCTV cameras and drones are just two possible applications for Surge but it's suitable for really any video streaming case where RTP / RTSP is possible.

Looking Outward

We have big plans for Surge and a lot of hard work has already been done but as software developers we understand the potential in putting your code in the hands of other developers. The Open-Source Software community knows only too well how many hands can make light work and a community of global contributors often improves a product in untold ways.

We're not quite there yet though – it's still early days for Surge and as such, it's still under wraps. However, we're keen to show it off so, here we'll give some insight into how simple and fast it is to integrate into an app on the platform of your choice.

Basic Setup

At its core, Surge is a video streaming library so any basic setup will require a video to stream, naturally. You may already have a drone or camera providing an RTSP stream but if not, it's easy to setup a video stream locally from a file via a Gstreamer server.

The initial process for adding the Surge framework will be familiar for most developers and it's easily integrated into existing and new projects alike as a self-contained framework without any further dependency requirements.

iOS – Swift

Let's start here with a blank iOS project in Xcode 8 using Swift 3.0, with just a single view:

From there, we add the Surge framework to the project and link it by copying the framework into the project and adding it as a dependency in the project settings.

It's that simple! Now we're ready to use Surge in our iOS app. All we need is a UIImageView into which to render the frames from the video and an instance of the SurgeRtspPlayer. The UIImageView here has been setup in a storyboard and linked to our view controller but it could easily be set up programmatically. The SurgeRtspPlayer can also take an optional delegate which we've set here to be the view controller itself, where we update an onscreen FPS counter.

class ViewController: UIViewController, SurgeRtspPlayerDelegate {

  @IBOutlet weak var videoStreamImageView: UIImageView!
  @IBOutlet weak var fpsLabel: UILabel!

  override func viewDidLoad() {
    super.viewDidLoad()
    let rtspPlayer: SurgeRtspPlayer = SurgeRtspPlayer()
    rtspPlayer.delegate = self
    rtspPlayer.playerView = videoStreamImageView

    if let url = URL(string: "rtsp://127.0.0.1:8554/test") {
      rtspPlayer.initiatePlaybackOf(url: url)
    }
  }

  func rtspPlayerDidObservePlaybackFrameRate(player: SurgeRtspPlayer, frameRate: UInt) {
    fpsLabel.text = "\(frameRate) fps"
  }
}

Android – Java

On Android, Surge is currently supported from API 19 (KitKat) and above. Here we're using Android Studio 2.3 and creating a basic, single-activity application:

Next we need to add Surge as a project dependency, which similarly easy to iOS. The Surge Android library is distributed as an aar archive and should be added to the /app/libs/ folder of the project.

From there, Gradle just needs some instructions to compile the library. Firstly, in the project's build.gradle we request that the libs folder be flattened:

allprojects {
    repositories {
        jcenter()
        flatDir {
            dirs 'libs'
        }
    }
}

And finally we can now compile the Surge library within the app's build.gradle file:

dependencies {
    compile(name: 'surge', ext: 'aar')
}

Once we have that in place, we can begin referencing the various Surge classes within our Activity. On Android, Surge renders into a Surface via a TextureView, which is again a little more complex than the iOS setup but Surge handily abstracts a lot of the finer details as usual. With a TextureView set up in our layout, its only necessary to add a listener for the textures availability and use the supplied SurfaceTexture to create a SurgeSurface. This is then passed to the player when initiating a stream.

(For demonstrative purposes we're using Java here, but it's worth noting here that Surge is fully supported in Kotlin too.)

public class MainActivity extends AppCompatActivity implements TextureView.SurfaceTextureListener, SurgeRtspPlayerDelegate  {

    private final SurgeRtspPlayer player = new SurgeRtspPlayer();
    private TextureView textureView;
    private TextView fpsLabel;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        textureView = (TextureView) findViewById(R.id.texture_view);
        textureView.setSurfaceTextureListener(this);
        fpsLabel = (TextView) findViewById(R.id.fps_label);
        player.delegate = this;
    }

    @Override
    public void onSurfaceTextureAvailable(SurfaceTexture surfaceTexture, int width, int height) {
        SurgeSurface surface = new SurgeSurface(new Surface(surfaceTexture), width, height);
        player.initiatePlaybackOf(
                "rtsp://127.0.0.1:8554/test",
                surface,
                new PlayerCallback() {
                    @Override
                    public void response(boolean result) {
                    }
                });
    }

    @Override
    public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
        player.stop();
        return false;
    }

    @Override
    public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
    }

    @Override
    public void onSurfaceTextureUpdated(SurfaceTexture surface) {
    }

    @Override
    public void rtspPlayerDidUpdateFps(int fps) {
        fpsLabel.setText(fps + " fps");
    }
}

Xamarin – C#

It's no secret that we're big fans of Xamarin here at Instil. In fact, the aforementioned application victor Go is built using Xamarin and leverages the power of Surge for both iOS and Android. As such, C# bindings for both platforms are available and are simple to integrate into a Xamarin solution. Examples here are shown using C# 7.0 in Visual Studio for Mac 7.2

For each platform, a dll library is supplied which packages the entire Surge framework along with a C# interface to the same APIs upon which we've already touched. Integrating this into a project is as simple as copying the dll to a local project folder and adding a '.Net assembly' reference within the solution.

In the case of both platforms, once the library has been successfully added, it can be imported into any classes within the solution and used in the same way described per platform above, albeit via a C# API.

For example, a typical Xamarin Android C# Activity for playback could look something like the following. It should be obvious that this an almost identical structure to the Android Java example shown earlier, but accessed here via a C# API, which serves to highlight the consistency of the Surge APIs across platforms, including its C# bindings for Xamarin. Developers familiar with native platform-specific paradigms should feel at ease using the Surge frameworks and associated APIs.

[Activity(Label = "Surge Rtsp Player")]
public class MainActivity : Activity, TextureView.ISurfaceTextureListener, ISurgeRtspPlayerDelegate {

    SurgeRtspPlayer rtspPlayer = new SurgeRtspPlayer();
    TextureView textureView;
    TextView fpsLabel;

    protected override void OnCreate(Bundle savedInstanceState) {
        base.OnCreate(savedInstanceState);
        SetContentView(Resource.Layout.Main);
        textureView = FindViewById<TextureView>(Resource.Id.texture_view);
        textureView.SurfaceTextureListener = this;
        fpsLabel = FindViewById<TextView>(Resource.Id.fps_label);
        rtspPlayer.Delegate = this;
    }

    public void OnSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
        var surgeSurface = new SurgeSurface(new Surface(surface), width, height);
        rtspPlayer.InitiatePlaybackOf(
            "rtsp://127.0.0.1:8554/test",
            surgeSurface,
            (bool obj) => { });
    }

    public bool OnSurfaceTextureDestroyed(SurfaceTexture surface) {
        return true;
    }

    public void OnSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
    }

    public void OnSurfaceTextureUpdated(SurfaceTexture surface) {
    }

    public void RtspPlayerDidUpdateFps(int fps) {
        fpsLabel.Text = $"{fps} fps";
    }
}

Other Features

We showed a basic Surge integration into three different types of app development but there are more advanced features available currently that we didn't cover. All examples here are provided in Swift but the APIs exist in all the current platform framework APIs.

UDP support

RTSP can be delivered over interleaved TCP or UDP. Surge by default will connect over TCP but it can be configured to use UDP. This is useful in situations where reduced latency is preferred over video stream integrity. Frames are not guaranteed in the correct order so some glitching can occur but the stream is lightning quick. Making the switch is as simple as turning off interleaved TCP on the player:

rtspPlayer.interleavedTcpTransport = false

Authentication

Some RTSP streams require a username and password and this basic authentication is available within Surge when initiating streams.

rtspPlayer.initiatePlaybackOf(url: url, username: "username", password: "passw0rd")

Seeking

Surge has full support for standard transport controls including play, stop and seek. Streams can be requested to start and end at a specific time, or moved to a specific time during playback, which is useful in siutations for recorded video streamed from a server.

rtspPlayer.initiatePlaybackOf(url: url, username: nil, password: nil, startDate: Date.init(timeIntervalSinceNow:-60), endDate: Date.init())

rtspPlayer.seekTo(startTime: Date.init(timeIntervalSinceNow:-60), endTime: Date.init())

Mutliple Streams

Due to the way that Surge has been built, it's possible to have multiple, simultaneous streams running. The maximum limit on this can vary depending on device capabilities but the more modern, beefy mobile hardware can run 9 or more concurrent streams. Most use cases, especially on phones, would rarely call for more than 4 but there's no hard limit set within the library itself so the responsibility falls on the devloper to choose a sensible number.

Taking advantage of Surge's concurrent streaming behaviour is as simple as creating a SurgeRtspPlayer instance for each stream:

rtspPlayer1.initiatePlaybackOf(url: url1, username: "username1", password: "passw0rd1")
rtspPlayer2.initiatePlaybackOf(url: url2, username: "username2", password: "passw0rd2")
rtspPlayer3.initiatePlaybackOf(url: url3, username: "username3", password: "passw0rd3")

How does that sound?

Are you building an app with video stream capabilities over RTSP? Perhaps you're avoiding it due to the lack of adequate libraries and/or suitable licensing. Or maybe you're already rolling your own solution for RTSP video streaming and would prefer a simple, quick-to-implement, no nonsense API with high performance streaming, so you can focus on the main features of your application. Whatever your use case, Surge may be right the solution for you. Hopefully we've shown here just how simple it can be to implement in your project and we'd be happy to give you a more in-depth demo of its full capabilities so feel free to get in touch!