Skip to main content
Intermediate For web developers whose page already shows a stream. After this page, your page and your Unreal application exchange messages in both directions. Your page and your Unreal application talk over the stream itself. No extra server is involved.

Two names for every call

If you learned Pixel Streaming from Epic’s material, you know emitUIInteraction, emitConsoleCommand and addResponseEventListener. They all work on the stream object. The SDK also has its own names for the same calls. Use whichever you know.
You can mix the two sets in one page. The rest of this page shows both in every example.

Why the SDK has its own names as well

Both sets are methods of the same stream object and do the same work. The SDK’s own names exist for three reasons.
  1. A return value you can act on. send() and consoleCommand() return true or false, so your page knows whether the message was handed to the stream. emitUIInteraction() and emitConsoleCommand() on the stream object return the same true or false, because they are the same calls.
  2. Listeners that survive a rebuilt connection. When a connection drops, the SDK can build a new one. Epic’s own object (stream.pixelStreaming) is replaced then. Listeners you add through the stream object stay attached: on(), addResponseEventListener() and addEventListener(). Listeners you add directly on stream.pixelStreaming are lost with the old object.
  3. Names that do not depend on Epic’s library. send(), consoleCommand() and on('ueMessage') are part of the SDK’s v2 API. Inside the SDK, one module talks to Epic’s library (at present Epic’s frontend library for Unreal Engine 5.7), and everything specific to Epic stays in that module. The SDK is built this way so that a rename in a later version of Epic’s library is handled there, and the SDK’s names stay as they are.
on('ueMessage') also works like every other SDK event: it returns a function that unsubscribes, it takes any number of listeners, and in TypeScript a typo in the event name does not compile.
The difference that matters is not between the two sets of names. It is between the stream object and Epic’s own object. Call either set of names on stream and you get the return value and the listeners that survive. stream.pixelStreaming is Epic’s real object: it is there when you need it, and it follows Epic’s library. See Pixel Streaming events and object.

What your Unreal app needs

Messages only do something when your application handles them. Set this up once in Unreal. The setup is the same for a share link, an iframe and the Web SDK.

Receive and send JSON in Unreal

The Blueprint for both directions, step by step.

Send files to the viewer

The Send File node, file types and file paths on a streaming server.

Send a message to Unreal

1

Wait until the stream is up

A message can only be sent while the state is streaming. Nothing is kept for later.
2

Send an object

The SDK turns it into JSON. In Unreal, Get Json String Value reads its fields.
3

Check the result

true: the message was handed to the stream. false: it was not sent. Send it again once the state is streaming.
With the Blueprint from the Unreal page in place, On Input Event fires in your application, and Get Json String Value with the field name action returns teleport.
You can also pass a string. It is sent as a JSON string, so an object is easier to read in Unreal.

When a message is not delivered

send() and emitUIInteraction() never throw and never queue. They tell you with the return value.
true means the message left your page. It is not a receipt from your application. If your page must know that the application acted, have the application answer with Send Pixel Streaming Response, and listen for that answer.
A fresh session means a new application process (see Reconnection and recovery). Messages you sent before are not replayed. Send the state your application needs again when streaming returns. This helper sends now when it can, and otherwise as soon as the stream is up:
Viewers of a shared session watch only. Send messages from the host’s page.
Messages to and from your application are not part of what the SDK reports to Streampixel.

Receive messages from Unreal

Your application sends text with Send Pixel Streaming Response. Your page receives exactly that text.
When your application runs Send Pixel Streaming Response with {"event":"doorOpened"}, the browser console shows From Unreal: and the object.
The two listener styles deliver the same messages. They differ in shape: Attach listeners right after await Streampixel.create(...). They are in place long before the first message can arrive.

Messages the SDK acts on for you

The SDK reads four kinds of message and acts on them, as Streampixel’s share page does. Each one still reaches your listeners.

Console commands

A console command is an Unreal console command, such as stat fps or t.MaxFPS 30, sent from your page.
The call returns false in the same situations as a message, and in two more:
On Streampixel’s servers every session is launched with -AllowPixelStreamingCommands: see Launch arguments and startup commands. When you test locally you start Unreal yourself, so check that your launch arguments include it.
Only commands your build understands do anything. A Shipping build has fewer console commands than a Development build.
For its own startup commands the SDK tries again every 500 ms for up to 10 seconds. This helper does the same for yours.
console-command-when-ready.js
The state returns to streaming after every reconnect, so the command runs again then. Prefer commands that are safe to repeat: stat fps is a toggle, t.MaxFPS 30 is not.
To change the resolution, use stream.setResolution(). It needs no console-command permission.

Launch parameters

Launch parameters are one message that the SDK sends for you when the application starts. Use them for what the application must know first: which unit to show, which language to use.
Launch parameters
A share link passes its URL query parameters to the application. To do the same in your page: launchParams: Object.fromEntries(new URLSearchParams(location.search)).
Launch parameters are a message. They are not Unreal command-line arguments. Command-line arguments are a project setting: see Launch arguments and startup commands.

Messages the SDK sends by itself

Besides launchParams, the SDK sends two things to your application on every video start. A viewer of a shared session sends neither.

Files from the application

Your application can send a file to the viewer with the Send File node: a screenshot, a report, a save file. The Unreal side is on Send files to the viewer.
{ received: number; total: number; name: string }
Fires while the file arrives. received and total count the pieces of the file, not bytes.
{ blob: Blob; name: string; mimeType: string }
Fires once, when the file is complete. name is download followed by the extension your application gave, for example download.png: Pixel Streaming sends a file’s type and extension, not its name. mimeType is application/octet-stream when the application gave none.
What happens depends on whether your page listens:
Your own Save button
Unreal’s Launch URL node opens a link on the server, not on the viewer’s device. To open a link for the viewer, the application sends the link as the whole text of a Send Pixel Streaming Response. The SDK treats a message as a link when it is one https:// address and nothing else. A message with any other scheme is not treated as a link.
{ url: string; open: () => void }
The application sent a link. open() opens it in a new tab. On phones and tablets, call open() inside a click or tap handler: those browsers refuse a tab opened in any other way.
A button for links
The link also reaches your ueMessage and response listeners, like every message. Open it in one place only.

Screenshots

There are two ways to take one. Both capture the current video frame, without anything your page draws on top.
captureScreenshot() returns the frame as a PNG data URL. It returns null before the first frame.
Screenshot button
For a render at a higher resolution than the stream, let the application create the image and send it as a file.

Pixel Streaming events and object

Code written against Epic’s library often listens to Epic’s events, or uses Epic’s PixelStreaming object directly. Both are available.
method
Subscribes to an event of Epic’s library by its original name, for example webRtcConnected, playStream or afkWarningActivate. The SDK attaches your listener again whenever it rebuilds the connection. Remove it with stream.removeEventListener(type, listener).
Epic's PixelStreaming object, or null
Epic’s own object for the current connection. It is null before a connection exists. It is replaced when the SDK rebuilds the connection, so read stream.pixelStreaming each time you use it and never keep it in a variable.Do not call its reconnect(), unmuteMicrophone() or unmuteCamera(): the SDK owns reconnection and capture.
Epic's event names on the stream object
In TypeScript, stream.pixelStreaming has the type unknown: the SDK does not export Epic’s classes. Cast it to the type from Epic’s package if you use that package’s types.

Troubleshooting

The video is not up. Check stream.state.kind: it must be streaming. Use the sendWhenStreaming helper on this page, or send from a state listener.
The message reached the stream. Check the Unreal side: the Pixel Streaming Input component is added, On Input Event is bound, and the field name in Get Json String Value matches the key you send. Print the descriptor in Unreal to see what arrives. The steps are on Unreal side.
Either the stream started a moment ago and the application has not reported its settings yet (try again for a few seconds), or the application was launched without -AllowPixelStreamingCommands.
That is how responses arrive: as the text your application sent. Parse it with JSON.parse() when it is JSON, as in the examples above.
You added it on stream.pixelStreaming. That object is replaced when the SDK rebuilds the connection. Add the listener on the stream object: stream.addResponseEventListener(), stream.addEventListener() or stream.on().
The SDK started a fresh session, which is a new application process. launchParams were sent again. Anything else your page sent before must be sent again: do it when the state becomes streaming.

Next steps

Controls and input

Sound, resolution, fullscreen, keyboard, touch, microphone and camera.

Events and states

One level up: every state and every event, with payloads.

Cookbook

Short recipes built on these calls.

Stream API reference

Every method and property of the stream object.