@@ -40,7 +40,7 @@ This post is intended to be a kind of an "unboxing" experience from the perspect
<h3><aname="what-is-webgl-streaming">What is WebGL streaming</a></h3>
If you read past blog-posts, you can just skip this section. I would however recommend to read the <ahref="https://doc-snapshots.qt.io/qt5-dev/webgl.html">documentation</a>at least (<i>the link will point to 5.12 docs after release</i>).
If you read past blog-posts, you can just skip this section. I would however recommend to at least read the <ahref="https://doc-snapshots.qt.io/qt5-dev/webgl.html">documentation</a> (<i>the link will point to 5.12 docs after release</i>).
WebGL streaming is a <ahref="http://doc.qt.io/qt-5/qpa.html">QPA plugin</a> that sends ("streams") OpenGL calls of your Qt Quick application over the network and in turn those are translated into <ahref="https://en.wikipedia.org/wiki/WebGL">WebGL</a> calls and thus can be rendered at <ahref="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API">HTML5 Canvas</a>. What it means in practice is that you can have an application running on a remote host and render its GUI in a local web-browser.
...
...
@@ -139,7 +139,7 @@ Let's now run it on Raspberry Pi using WebGL streaming plugin, connect to it fro
If video doesn't play in your browser, you can download it <ahref="https://git.qt.io/arsidyak/webgl-release/raw/master/webgl-release-blog/img/webgl-different-canvas-sizes.mp4">here</a></div>
As you can see, when I resize the browser window, application (<i>beside nicely adapting its layout</i>) reports changed screen resolution values, which wasn't the case when I ran it without WebGL enabled. The reason for that is because it takes canvas dimensions for the screen resolution. Here are they in web-browser inspector:
As you can see, when I resize the browser window, application (<i>beside nicely adapting its layout</i>) reports changed screen resolution values. This is because it takes canvas dimensions for the screen resolution. Let's check that in web-browser inspector:
@@ -151,13 +151,13 @@ So application does indeed run on Raspberry, and what I have in my browser is ju
<h4><aname="demo-camera">Camera</a></h4>
This demo has some actual practical usage - it's a camera controlled by robotic-ish arm which is mounted on a Raspberry Pi device:
This demo is a bit more practical one - it's a camera controlled by robotic-ish arm which is mounted on a Raspberry Pi device:
<imgclass="aligncenter"src="https://qt-blog-uploads.s3.amazonaws.com/wp-content/uploads/2018/11/rpi-camera-pimoroni-hat.jpg"title="Raspberry Pi with camera and Pimoroni pan-tilt HAT"/>
The idea is to control the camera (its pan and tilt) with a Qt-based application running on the device, but to do that remotely from a web-browser on some tablet. And of course, we would like to see what cameras is looking at (to stream its video-stream). Would be also nice to be able make photos and preview those as well.
The idea is to control the camera (its pan and tilt) with a Qt-based application running on the device, but to do that remotely from a web-browser on some tablet. And of course we would like to see what cameras is looking at (<i>its viewfinder</i>). And it also would be nice to make photos with the camera.
Here's a list of the required hardware for such a setup:
Here's a list of required hardware for such a setup:
@@ -170,7 +170,7 @@ And that's <a href="https://learn.pimoroni.com/tutorial/sandyj/assembling-pan-ti
Now let's take a little detour (<i>spoiler: I will be promoting Qt's commercial features</i>). You might have noticed that for <ahref="#demo-device-information">Device Information</a> demo I used <ahref="https://doc.qt.io/QtForDeviceCreation/qtb2-index.html">Boot to Qt</a> image, saving myself quite some time and efforts with regards to building Qt-based application for Raspberry Pi and deploying it there.
But Boot to Qt is a commercial-only feature, and without it you'll have to go though some more steps while setting up system environment. Here's what it takes with a regular <ahref="https://www.raspberrypi.org/downloads/raspbian/">Raspbian Stretch Lite</a> image as an example:
But Boot to Qt is a commercial-only feature, and without it you'll have to go though some more steps setting up system environment. Here's what it takes with a regular <ahref="https://www.raspberrypi.org/downloads/raspbian/">Raspbian Stretch Lite</a> image as an example:
<ul>
<li>Since Qt Multimedia module is used, you need to make sure that <ahref="https://en.wikipedia.org/wiki/GStreamer">GStreamer</a> is installed in the system and you have correct plugins available;</li>
...
...
@@ -179,13 +179,13 @@ But Boot to Qt is a commercial-only feature, and without it you'll have to go th
<li>Come up with a convenient way of building/deploying your applications on device.</li>
</ul>
Even though the list of steps is not too long, in practice it can take you up to several days before you get a working setup, whether with Boot to Qt image you get everything working out-of-the-box, and you can run your applications on the connected device right from the Qt Creator. But enough with the promotional part, let's get back to demo.
Even though this list of steps is not too long, in practice it can take you up to several days before you get a working setup, whether with Boot to Qt image you get everything working out-of-the-box, and you can run your applications on the connected device right from the Qt Creator. But enough with the promotional part, let's get back to demo.
The GUI layout looks like the following:
<imgclass="aligncenter"src="https://qt-blog-uploads.s3.amazonaws.com/wp-content/uploads/2018/11/webgl-release-mac-camera.png"title="Qt Multimedia, camera demo application, viewfinder"/>
Most of the space is taken by the camera's viewfinder, which is implemented with <ahref="http://doc.qt.io/qt-5/qml-qtmultimedia-videooutput.html">VideoOutput</a> and <ahref="http://doc.qt.io/qt-5/qml-qtmultimedia-camera.html">Camera</a> itself:
Most of the space on the first tab is taken by the camera's viewfinder, which is implemented with <ahref="http://doc.qt.io/qt-5/qml-qtmultimedia-videooutput.html">VideoOutput</a> and <ahref="http://doc.qt.io/qt-5/qml-qtmultimedia-camera.html">Camera</a> itself:
<preclass="brush:bash">Camera { id: camera }
...
...
@@ -215,7 +215,7 @@ There are two <a href="https://doc.qt.io/qt-5.11/qml-qtquick-controls2-slider.ht
}
</pre>
Pan-Tilt HAT servos are interfaced via <ahref="https://en.wikipedia.org/wiki/I²C">I2C</a>, and due to the lack of time I went with a fast and dirty solution - by using Pimoroni's <ahref="https://github.com/pimoroni/pantilt-hat">Python library</a>. At some point I would like to do it properly with C/C++, although it's really not the point of the demo.
Pan-Tilt HAT servos are interfaced via <ahref="https://en.wikipedia.org/wiki/I²C">I2C</a>, and due to the lack of time I went with fast and dirty solution - by using Pimoroni's <ahref="https://github.com/pimoroni/pantilt-hat">Python library</a>. At some point I would like to do it properly with C/C++, although it's really not the point of the demo.
There is also a <ahref="https://doc.qt.io/qt-5/qml-qtquick-controls2-button.html">button</a> for taking photos using <ahref="http://doc.qt.io/qt-5/qml-qtmultimedia-cameracapture.html">CameraCapture</a>:
...
...
@@ -241,7 +241,7 @@ Second tab contains a list of taken photos:
<imgclass="aligncenter"src="https://qt-blog-uploads.s3.amazonaws.com/wp-content/uploads/2018/11/webgl-release-mac-photos.png"title="Qt Multimedia, camera demo application, photos"/>
List of files is provided by <ahref="http://doc.qt.io/qt-5/qml-qt-labs-folderlistmodel-folderlistmodel.html">FolderListModel</a>:
...which is implemented by <ahref="http://doc.qt.io/qt-5/qml-qt-labs-folderlistmodel-folderlistmodel.html">FolderListModel</a>:
<preclass="brush:bash">ListView {
FolderListModel {
...
...
@@ -259,52 +259,52 @@ List of files is provided by <a href="http://doc.qt.io/qt-5/qml-qt-labs-folderli
Full application source code is available <ahref="https://git.qt.io/arsidyak/webgl-release/tree/master/webgl-release-demo">here</a>.
Now let's see it in action. There are 3 spirits placed on my table, surrounding the camera, and I want to take photos of each. I built and ran the application on device with <code>-platform webgl</code>, and connected to it from Safari on my iPad:
Now let's see it in action. There are 3 spirits placed on my table, surrounding the camera, and I want to take photos of each. I built and ran the application on device with <code>-platform webgl</code>, and connected to it over Wi-Fi from Safari on my iPad:
As you can see, original plan worked out nicely: seeing camera's viewfinder, I can remotely control its position precisely and take photos of the objects I'm interested in.
As you can see, the plan worked out just fine: seeing camera's viewfinder, I can remotely control its position and take photos of the objects I'm interested in.
<h3><aname="use-cases">Use cases</a></h3>
Most obvious use case for WebGL streaming is the ability to have a decent GUI for some low-end device with limited computing power, without GPU, and quite often without any display at all. For instance, that is a common scenario for industrial automation domain, where you can have lots of headless devices installed all over the factory: they can be distributed over quite a significant area or even mounted in places with hazardous environment - being able to control/configure those remotely comes to be rather handy.
Reading discussion at <ahref="https://news.ycombinator.com/item?id=14718043">Hacker News</a>, I stumbled upon a "reverse" idea: what if it's the other way around, what if "device" is actually a very powerful server, and you work with it from your regular desktop. That way you can can perform some heavy calculations on the server having its GUI in your web-browser (<i>which actually begs for an HTML-based frontend but more on this in the <ahref="#webgl-streaming-vs-actual-web">next section</a></i>).
Reading discussion at <ahref="https://news.ycombinator.com/item?id=14718043">Hacker News</a>, I stumbled upon a "reverse" idea: what if it's the other way around, what if "device" is actually a very powerful server, and you work with it from your regular desktop. That way you can can perform some heavy calculations on the server while having GUI in your web-browser (<i>which actually begs for an HTML-based frontend but more on this in the <ahref="#webgl-streaming-vs-actual-web">next section</a></i>).
Another possible use-case is an anti-piracy measure. Let's say you want to protect your software from being "cracked" or "pirated". Obviously, if there is nothing running on the client, then there is nothing to crack as your users only have GUI rendered in their browsers, and the application itself is running on your server. Sounds interesting, but there are several drawbacks here:
<ul>
<li>While WebGL streaming performs well in local network, using it over the internet will result in significant latency;</li>
<li>Connection is not encrypted, so it is not secure. Although this functionality is still in research;</li>
<li>Connection is not encrypted, so it is not secure;</li>
<li>Currently only one connection at a time is supported (<i>so only one user</i>).</li>
</ul>
Overall, supporting only one connection fairly reduces the number of possible use cases, and unfortunately it is unlikely that current implementation will improve in that regard as it is more of a task for <ahref="https://bugreports.qt.io/browse/QTBUG-62425">Qt 6</a>. By the way, there is an idea to complement streaming with an ability of mirroring as in some cases the latter is needed more.
Overall, supporting only one connection at a time fairly reduces the number of possible use cases, and unfortunately it is unlikely that current implementation of the feature will improve in that regard, so it is more of a task for <ahref="https://bugreports.qt.io/browse/QTBUG-62425">Qt 6</a>. By the way, there is an idea to complement streaming with an ability of mirroring as in some cases having the latter is more important.
Speaking about mirroring, I would also like to mention our recent <ahref="https://www.qt.io/events/remote-ui-with-qt-for-automation-on-arm-based-edge-devices-1539882056/">webinar</a> we had together with Toradex. There you can see an interesting combination of WebGL streaming and <ahref="https://doc.qt.io/qt-5/qtremoteobjects-index.html">Remote Objects</a>, which allows you to implement mirroring functionality as of now already.
Speaking about mirroring, I would like to mention our recent <ahref="https://www.qt.io/events/remote-ui-with-qt-for-automation-on-arm-based-edge-devices-1539882056/">webinar</a>that we had together with Toradex. There you can see an interesting combination of WebGL streaming and <ahref="https://doc.qt.io/qt-5/qtremoteobjects-index.html">Remote Objects</a>, which allows you to implement mirroring functionality as of now already.
Another noticeable aspect is so-called "zero install" concept - you don't have to install/deploy anything on clients (<i>desktops/tablets/smartphones/etc</i>), the only thing needed is a web-browser. However, <ahref="http://blog.qt.io/blog/2018/11/19/getting-started-qt-webassembly/">WebAssembly</a> seems to be a bit more suitable for that purpose.
Another noticeable aspect of WebGL streaming is so-called "zero install" concept - you don't have to install/deploy anything on clients (<i>desktops/tablets/smartphones/etc</i>) as the only thing needed is just a web-browser. However, <ahref="http://blog.qt.io/blog/2018/11/19/getting-started-qt-webassembly/">Qt for WebAssembly</a> seems to be a bit more suitable for that purpose.
<h4><aname="webgl-streaming-vs-actual-web">WebGL streaming vs actual web</a></h4>
Some of you might ask, what is the point of relying on WebGL streaming in the first place? Since it's all about web-browser, then one can just take a regular web-server and create a web-application - result will be almost the same: backend is hosted on the remote device and HTML-based GUI is rendered in the web-browser.
Some of you might ask, what is the point of relying on WebGL streaming in the first place? Since it's all about web-browser, one can just take a regular web-server and create a web-application - result will be almost the same: backend is hosted on the remote device and HTML-based GUI is rendered in the web-browser.
That is a very good an fair question. I actually have some experience in web-development, so I asked this question myself. And here's my answer, which hopefully won't start yet another holy war.
That is a very good an fair question. I actually have some experience in web-development, so I asked this question myself. So let's try to answer it, hopefully without starting yet another holy war.
Indeed, in some cases it is enough just to have a simple REST API, especially if you only need to get some data from devices. So it is likely that Qt-based application with WebGL streaming will be an overkill here.
Indeed, in some cases it is enough just to have a simple REST API, especially if you only need to get some plain text data values. So it is likely that Qt-based application with WebGL streaming would be an overkill for such purpose.
However, in more sophisticated scenarios, where, for example, you need to work with some hardware, Qt-based application fits better, and WebGLstreaming will take care of the client part. I would also mention that creating a complex, appealing and performant frontend is (considerably) easier with Qt Quick rather than with HTML/CSS/JS, but this statement does look like a beginning of yet another holy war, so I'll keep that as my personal opinion.
However, in more sophisticated scenarios (<i>for example, when you need to control some hardware</i>) Qt-based application with WebGL-streamed GUI might fit better. I would also mention that creating a complex, appealing and performant frontend is (considerably) easier with Qt Quick rather than with HTML/CSS/JS, but this statement does look like a beginning of yet another holy war, so I'll keep that as my personal opinion.
And the last thing worth to mention here - if you already have a Qt-based application, then WebGL streaming is an obvious option, because it will cost you nothing to have a remote GUI for that application.
And the last thing worth to mention here - if you already have a Qt-based application, then WebGL streaming is an obvious option, because it will cost you nothing to have a remote GUI for it.
WebGL streaming plugin is available under commercial and Open Source licenses (<i>GPLv3 only</i>). For commercial customers it is included in both Application Development and Device Creation products with no additional charge.
WebGL streaming plugin is available under commercial and Open Source licenses (<i>but GPLv3 only</i>). And for commercial customers it is included in both Application Development and Device Creation products with no additional charge.
<h3><aname="conclusion">Conclusion</a></h3>
So you're now able to use web-browser as a remote GUI client for your Qt Quick applications with no efforts - it only takes one command line parameter.
In terms of further development, I reckon the next thing to be expected is connection security/encryption, both for WebSocket and WebServer. WebSocket part should be pretty straightforward as <ahref="http://doc.qt.io/qt-5/qwebsocket.html">QWebSocket</a> already supports secure connection (<code>wss://</code>). When it comes to WebServer part, if you remember, from the very beginning it was a temporary solution, and research on proper implementation is still ongoing.
In terms of further development, I reckon the next thing to be expected is connection security/encryption, both for WebSocket and WebServer. WebSocket part should be pretty straightforward as <ahref="http://doc.qt.io/qt-5/qwebsocket.html">QWebSocket</a> already supports secure connection (<code>wss://</code>). And WebServer part, if you remember, from the very beginning was a temporary solution, and research on proper implementation (<i>including support for HTTPS</i>) is still ongoing.
Meanwhile, if you have any other feature-requests or maybe bugs to report, please use our tracker for that: <ahref="http://bugreports.qt.io/">http://bugreports.qt.io/</a> (choose <b>QPA: WebGL</b> component). Your feedback will help our product management team to shape roadmap of the feature.
Meanwhile, if you have any other feature-requests or maybe bugs to report, please use our tracker for that: <ahref="http://bugreports.qt.io/">http://bugreports.qt.io/</a> (<i>choose <b>QPA: WebGL</b> component</i>). Your feedback will help our product management team to shape the feature's roadmap.
