Two names for every call
If you learned Pixel Streaming from Epic’s material, you knowemitUIInteraction, emitConsoleCommand and addResponseEventListener. They all work on the stream object. The SDK also has its own names for the same calls. Use whichever you know.
You can mix the two sets in one page. The rest of this page shows both in every example.
Why the SDK has its own names as well
Both sets are methods of the samestream object and do the same work. The SDK’s own names exist for three reasons.
- A return value you can act on.
send()andconsoleCommand()returntrueorfalse, so your page knows whether the message was handed to the stream.emitUIInteraction()andemitConsoleCommand()on the stream object return the sametrueorfalse, because they are the same calls. - Listeners that survive a rebuilt connection. When a connection drops, the SDK can build a new one. Epic’s own object (
stream.pixelStreaming) is replaced then. Listeners you add through the stream object stay attached:on(),addResponseEventListener()andaddEventListener(). Listeners you add directly onstream.pixelStreamingare lost with the old object. - Names that do not depend on Epic’s library.
send(),consoleCommand()andon('ueMessage')are part of the SDK’s v2 API. Inside the SDK, one module talks to Epic’s library (at present Epic’s frontend library for Unreal Engine 5.7), and everything specific to Epic stays in that module. The SDK is built this way so that a rename in a later version of Epic’s library is handled there, and the SDK’s names stay as they are.
on('ueMessage') also works like every other SDK event: it returns a function that unsubscribes, it takes any number of listeners, and in TypeScript a typo in the event name does not compile.
The difference that matters is not between the two sets of names. It is between the stream object and Epic’s own object. Call either set of names on
stream and you get the return value and the listeners that survive. stream.pixelStreaming is Epic’s real object: it is there when you need it, and it follows Epic’s library. See Pixel Streaming events and object.What your Unreal app needs
Messages only do something when your application handles them. Set this up once in Unreal. The setup is the same for a share link, an iframe and the Web SDK.Receive and send JSON in Unreal
The Blueprint for both directions, step by step.
Send files to the viewer
The Send File node, file types and file paths on a streaming server.
Send a message to Unreal
1
Wait until the stream is up
A message can only be sent while the state is
streaming. Nothing is kept for later.2
Send an object
The SDK turns it into JSON. In Unreal, Get Json String Value reads its fields.
3
Check the result
true: the message was handed to the stream. false: it was not sent. Send it again once the state is streaming.With the Blueprint from the Unreal page in place, On Input Event fires in your application, and Get Json String Value with the field name
action returns teleport.When a message is not delivered
send() and emitUIInteraction() never throw and never queue. They tell you with the return value.
A fresh session means a new application process (see Reconnection and recovery). Messages you sent before are not replayed. Send the state your application needs again when
streaming returns.
This helper sends now when it can, and otherwise as soon as the stream is up:
Viewers of a shared session watch only. Send messages from the host’s page.
Messages to and from your application are not part of what the SDK reports to Streampixel.
Receive messages from Unreal
Your application sends text with Send Pixel Streaming Response. Your page receives exactly that text.When your application runs Send Pixel Streaming Response with
{"event":"doorOpened"}, the browser console shows From Unreal: and the object.
Attach listeners right after
await Streampixel.create(...). They are in place long before the first message can arrive.
Messages the SDK acts on for you
The SDK reads four kinds of message and acts on them, as Streampixel’s share page does. Each one still reaches your listeners.Console commands
A console command is an Unreal console command, such asstat fps or t.MaxFPS 30, sent from your page.
false in the same situations as a message, and in two more:
On Streampixel’s servers every session is launched with
-AllowPixelStreamingCommands: see Launch arguments and startup commands. When you test locally you start Unreal yourself, so check that your launch arguments include it.Send a command as soon as the application accepts it
Send a command as soon as the application accepts it
For its own startup commands the SDK tries again every 500 ms for up to 10 seconds. This helper does the same for yours.The state returns to
console-command-when-ready.js
streaming after every reconnect, so the command runs again then. Prefer commands that are safe to repeat: stat fps is a toggle, t.MaxFPS 30 is not.Launch parameters
Launch parameters are one message that the SDK sends for you when the application starts. Use them for what the application must know first: which unit to show, which language to use.Launch parameters
Launch parameters are a message. They are not Unreal command-line arguments. Command-line arguments are a project setting: see Launch arguments and startup commands.
Messages the SDK sends by itself
BesideslaunchParams, the SDK sends two things to your application on every video start. A viewer of a shared session sends neither.
Files from the application
Your application can send a file to the viewer with the Send File node: a screenshot, a report, a save file. The Unreal side is on Send files to the viewer.Fires while the file arrives.
received and total count the pieces of the file, not bytes.Fires once, when the file is complete.
name is download followed by the extension your application gave, for example download.png: Pixel Streaming sends a file’s type and extension, not its name. mimeType is application/octet-stream when the application gave none.Your own Save button
Links from the application
Unreal’sLaunch URL node opens a link on the server, not on the viewer’s device. To open a link for the viewer, the application sends the link as the whole text of a Send Pixel Streaming Response.
The SDK treats a message as a link when it is one https:// address and nothing else. A message with any other scheme is not treated as a link.
The application sent a link.
open() opens it in a new tab. On phones and tablets, call open() inside a click or tap handler: those browsers refuse a tab opened in any other way.A button for links
The link also reaches your
ueMessage and response listeners, like every message. Open it in one place only.Screenshots
There are two ways to take one. Both capture the current video frame, without anything your page draws on top.- Your page takes it
- Your application asks for it
captureScreenshot() returns the frame as a PNG data URL. It returns null before the first frame.Screenshot button
Pixel Streaming events and object
Code written against Epic’s library often listens to Epic’s events, or uses Epic’sPixelStreaming object directly. Both are available.
Subscribes to an event of Epic’s library by its original name, for example
webRtcConnected, playStream or afkWarningActivate. The SDK attaches your listener again whenever it rebuilds the connection. Remove it with stream.removeEventListener(type, listener).Epic’s own object for the current connection. It is
null before a connection exists. It is replaced when the SDK rebuilds the connection, so read stream.pixelStreaming each time you use it and never keep it in a variable.Do not call its reconnect(), unmuteMicrophone() or unmuteCamera(): the SDK owns reconnection and capture.Epic's event names on the stream object
In TypeScript,
stream.pixelStreaming has the type unknown: the SDK does not export Epic’s classes. Cast it to the type from Epic’s package if you use that package’s types.Troubleshooting
send() or emitUIInteraction() returns false
send() or emitUIInteraction() returns false
The video is not up. Check
stream.state.kind: it must be streaming. Use the sendWhenStreaming helper on this page, or send from a state listener.It returns true and nothing happens in Unreal
It returns true and nothing happens in Unreal
The message reached the stream. Check the Unreal side: the Pixel Streaming Input component is added, On Input Event is bound, and the field name in Get Json String Value matches the key you send. Print the descriptor in Unreal to see what arrives. The steps are on Unreal side.
consoleCommand() or emitConsoleCommand() returns false while the stream is up
consoleCommand() or emitConsoleCommand() returns false while the stream is up
Either the stream started a moment ago and the application has not reported its settings yet (try again for a few seconds), or the application was launched without
-AllowPixelStreamingCommands.My listener gets a string, not an object
My listener gets a string, not an object
That is how responses arrive: as the text your application sent. Parse it with
JSON.parse() when it is JSON, as in the examples above.My listener stopped after a reconnect
My listener stopped after a reconnect
You added it on
stream.pixelStreaming. That object is replaced when the SDK rebuilds the connection. Add the listener on the stream object: stream.addResponseEventListener(), stream.addEventListener() or stream.on().A link or a file does nothing on a phone
A link or a file does nothing on a phone
Phones and tablets open a tab or save a file only from a tap. Add an
openUrl or file listener that shows a button, or mount @streampixel/ui.The application lost its state after a pause in the video
The application lost its state after a pause in the video
The SDK started a fresh session, which is a new application process.
launchParams were sent again. Anything else your page sent before must be sent again: do it when the state becomes streaming.Next steps
Controls and input
Sound, resolution, fullscreen, keyboard, touch, microphone and camera.
Events and states
One level up: every state and every event, with payloads.
Cookbook
Short recipes built on these calls.
Stream API reference
Every method and property of the stream object.