> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streampixel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Talk to your Unreal app

> Send JSON messages and console commands from your web page to your Unreal Engine application, and receive its replies, files, links and screenshots, with the Streampixel Web SDK.

Intermediate For web developers whose page already shows a stream. After this page, your page and your Unreal application exchange messages in both directions.

Your page and your Unreal application talk over the stream itself. No extra server is involved.

## Two names for every call

If you learned Pixel Streaming from Epic's material, you know `emitUIInteraction`, `emitConsoleCommand` and `addResponseEventListener`. They all work on the stream object. The SDK also has its own names for the same calls. Use whichever you know.

```js title="Epic's names" theme={"dark"}
// Page to Unreal
stream.emitUIInteraction({ action: 'setColor', value: 'red' });
stream.emitConsoleCommand('stat fps');

// Unreal to page
stream.addResponseEventListener('my-listener', (text) => {
  console.log('Unreal says:', text);
});
```

```js title="Streampixel's names" theme={"dark"}
// Page to Unreal
stream.send({ action: 'setColor', value: 'red' });
stream.consoleCommand('stat fps');

// Unreal to page
stream.on('ueMessage', (message) => {
  console.log('Unreal says:', message);
});
```

| Epic's name | Streampixel's name | Relation |
| - | - | - |
| `stream.emitUIInteraction(data)` | `stream.send(data)` | The same call. Both return `true` or `false`. |
| `stream.emitConsoleCommand(cmd)` | `stream.consoleCommand(cmd)` | The same call. Both return `true` or `false`. |
| `stream.addResponseEventListener(name, fn)` | `stream.on('ueMessage', fn)` | The same messages. The listener styles differ: see [Receive messages](#receive-messages-from-unreal). |
| `stream.removeResponseEventListener(name)` | the function that `on()` returned, or `stream.off('ueMessage', fn)` | Stop listening. |
| `stream.addEventListener(type, fn)` | `stream.on(event, fn)` | Epic's events by Epic's names, or the SDK's events. See [Pixel Streaming events and object](#pixel-streaming-events-and-object). |

You can mix the two sets in one page. The rest of this page shows both in every example.

### Why the SDK has its own names as well

Both sets are methods of the same `stream` object and do the same work. The SDK's own names exist for three reasons.

1. **A return value you can act on.** `send()` and `consoleCommand()` return `true` or `false`, so your page knows whether the message was handed to the stream. `emitUIInteraction()` and `emitConsoleCommand()` on the stream object return the same `true` or `false`, because they are the same calls.
2. **Listeners that survive a rebuilt connection.** When a connection drops, the SDK can build a new one. Epic's own object (`stream.pixelStreaming`) is replaced then. Listeners you add through the stream object stay attached: `on()`, `addResponseEventListener()` and `addEventListener()`. Listeners you add directly on `stream.pixelStreaming` are lost with the old object.
3. **Names that do not depend on Epic's library.** `send()`, `consoleCommand()` and `on('ueMessage')` are part of the SDK's v2 API. Inside the SDK, one module talks to Epic's library (at present Epic's frontend library for Unreal Engine 5.7), and everything specific to Epic stays in that module. The SDK is built this way so that a rename in a later version of Epic's library is handled there, and the SDK's names stay as they are.

`on('ueMessage')` also works like every other SDK event: it returns a function that unsubscribes, it takes any number of listeners, and in TypeScript a typo in the event name does not compile.

> [!NOTE]
>
> The difference that matters is not between the two sets of names. It is between the **stream object** and **Epic's own object**. Call either set of names on `stream` and you get the return value and the listeners that survive. `stream.pixelStreaming` is Epic's real object: it is there when you need it, and it follows Epic's library. See [Pixel Streaming events and object](#pixel-streaming-events-and-object).

## What your Unreal app needs

Messages only do something when your application handles them. Set this up once in Unreal. The setup is the same for a share link, an iframe and the Web SDK.

> [card] [Receive and send JSON in Unreal](https://docs.streampixel.io/resources/json-message-communication/unreal-side-sending-and-receiving-json.md)
>
> The Blueprint for both directions, step by step.

> [card] [Send files to the viewer](https://docs.streampixel.io/resources/additional-unreal-engine-features/send-files-to-the-viewer.md)
>
> The **Send File** node, file types and file paths on a streaming server.

| Direction | In Unreal |
| - | - |
| Page to Unreal | Add the **Pixel Streaming Input** component (with the Pixel Streaming 2 plugin: **PixelStreaming2Input**). Bind **On Input Event**. Read a field with **Get Json String Value**. |
| Unreal to page | Call the **Send Pixel Streaming Response** node. Put your text, preferably JSON, in its descriptor field. |

## Send a message to Unreal

**1. Wait until the stream is up**

A message can only be sent while the state is `streaming`. Nothing is kept for later.

**2. Send an object**

The SDK turns it into JSON. In Unreal, **Get Json String Value** reads its fields.

```js title="Epic's names" theme={"dark"}
const sent = stream.emitUIInteraction({ action: 'teleport', to: 'lobby' });
```

```js title="Streampixel's names" theme={"dark"}
const sent = stream.send({ action: 'teleport', to: 'lobby' });
```

**3. Check the result**

`true`: the message was handed to the stream. `false`: it was not sent. Send it again once the state is `streaming`.

> [!TIP]
>
> With the Blueprint from the Unreal page in place, **On Input Event** fires in your application, and **Get Json String Value** with the field name `action` returns `teleport`.

You can also pass a string. It is sent as a JSON string, so an object is easier to read in Unreal.

### When a message is not delivered

`send()` and `emitUIInteraction()` never throw and never queue. They tell you with the return value.

| Situation | Return value | What to do |
| - | - | - |
| The state is `idle`, `awaitingPassword`, `connecting` or `queued`: the video is not up yet | `false` | Wait for `streaming`. |
| The state is `reconnecting` or `recovering`: the connection is being restored | Usually `false`. Treat any result in these states as "not sent". | Wait for `streaming`. |
| The state is `ended`, or you called `disconnect()` or `destroy()` | `false` | Create a new stream. |
| The state is `streaming` | `true` | Nothing. |
| Sent, and your application has no Blueprint that handles it | `true` | Bind **On Input Event** in Unreal. |

> [!WARNING]
>
> `true` means the message left your page. It is not a receipt from your application. If your page must know that the application acted, have the application answer with **Send Pixel Streaming Response**, and listen for that answer.

A fresh session means a new application process (see [Reconnection and recovery](https://docs.streampixel.io/resources/web-sdk-v2/events-and-states.md#reconnection-and-recovery)). Messages you sent before are not replayed. Send the state your application needs again when `streaming` returns.

This helper sends now when it can, and otherwise as soon as the stream is up:

```js title="Epic's names" theme={"dark"}
function sendWhenStreaming(stream, payload) {
  if (stream.emitUIInteraction(payload)) return;
  const off = stream.on('state', (s) => {
    if (s.kind === 'streaming' && stream.emitUIInteraction(payload)) off();
  });
}
```

```js title="Streampixel's names" theme={"dark"}
function sendWhenStreaming(stream, payload) {
  if (stream.send(payload)) return;
  const off = stream.on('state', (s) => {
    if (s.kind === 'streaming' && stream.send(payload)) off();
  });
}
```

> [!NOTE]
>
> Viewers of a [shared session](https://docs.streampixel.io/resources/web-sdk-v2/shared-sessions.md) watch only. Send messages from the host's page.

> [!NOTE]
>
> Messages to and from your application are not part of what the SDK reports to Streampixel.

## Receive messages from Unreal

Your application sends text with **Send Pixel Streaming Response**. Your page receives exactly that text.

```js title="Epic's names" theme={"dark"}
stream.addResponseEventListener('my-listener', (text) => {
  // text is always a string
  const data = text.trim().startsWith('{') ? JSON.parse(text) : text;
  console.log('From Unreal:', data);
});

// later
stream.removeResponseEventListener('my-listener');
```

```js title="Streampixel's names" theme={"dark"}
const off = stream.on('ueMessage', (message) => {
  // usually a string: parse it when it is JSON
  const data = typeof message === 'string' && message.trim().startsWith('{')
    ? JSON.parse(message)
    : message;
  console.log('From Unreal:', data);
});

// later
off();
```

> [!TIP]
>
> When your application runs **Send Pixel Streaming Response** with `{"event":"doorOpened"}`, the browser console shows `From Unreal:` and the object.

The two listener styles deliver the same messages. They differ in shape:

| | `addResponseEventListener(name, fn)` | `on('ueMessage', fn)` |
| - | - | - |
| What the listener gets | A string, always. A message that is not a string is turned into JSON text. | The message as it arrived (`unknown` in TypeScript). In practice a string. |
| Number of listeners | One per `name`. A second call with the same `name` replaces the first. | Any number. |
| Stop listening | `removeResponseEventListener(name)` | The function `on()` returned, or `off('ueMessage', fn)`. |
| A rebuilt connection | Stays attached. | Stays attached. |
| The session ends | Nothing more arrives. | Removed with all other `on()` listeners, after `ended` has fired. |

Attach listeners right after `await Streampixel.create(...)`. They are in place long before the first message can arrive.

### Messages the SDK acts on for you

The SDK reads four kinds of message and acts on them, as Streampixel's share page does. Each one **still reaches your listeners**.

| Message from your application | What the SDK does |
| - | - |
| `{ "message": { "type": "heartbeat" } }` | Cancels an idle countdown that is running. Send it during cutscenes, when the viewer is watching and not clicking. |
| `{ "type": "togglehoveringmouse", "value": true }`, also inside `{ "message": … }` | Switches hover mouse on or off. See [Controls and input](https://docs.streampixel.io/resources/web-sdk-v2/controls-and-input.md#hover-mouse). |
| `requestScreenshot`, also as `{ "message": "requestScreenshot" }` | Captures the current frame. See [Screenshots](#screenshots). |
| Text that is nothing but one `https://` link | Offers the link to the viewer. See [Links from the application](#links-from-the-application). |

## Console commands

A console command is an Unreal console command, such as `stat fps` or `t.MaxFPS 30`, sent from your page.

```js title="Epic's names" theme={"dark"}
const sent = stream.emitConsoleCommand('t.MaxFPS 30');
```

```js title="Streampixel's names" theme={"dark"}
const sent = stream.consoleCommand('t.MaxFPS 30');
```

The call returns `false` in the same situations as a message, and in two more:

| Situation | Return value |
| - | - |
| The application was not launched with `-AllowPixelStreamingCommands` | `false`, always |
| The stream has only just started and the application has not yet reported its settings | `false` for a moment, then `true` |

> [!NOTE]
>
> On Streampixel's servers every session is launched with `-AllowPixelStreamingCommands`: see [Launch arguments and startup commands](https://docs.streampixel.io/resources/additional-unreal-engine-features/launch-arguments-and-startup-commands.md). When you [test locally](https://docs.streampixel.io/resources/web-sdk-v2/test-locally.md) you start Unreal yourself, so check that your launch arguments include it.

Only commands your build understands do anything. A Shipping build has fewer console commands than a Development build.

> [accordion] Send a command as soon as the application accepts it
>
> For its own startup commands the SDK tries again every 500 ms for up to 10 seconds. This helper does the same for yours.
>
> ```js title="console-command-when-ready.js" theme={"dark"}
> function consoleCommandWhenReady(stream, command, waitedMs = 0) {
>   if (stream.consoleCommand(command)) return;   // or stream.emitConsoleCommand(command)
>   if (waitedMs >= 10_000 || stream.state.kind === 'ended') return;
>   setTimeout(() => consoleCommandWhenReady(stream, command, waitedMs + 500), 500);
> }
>
> stream.on('state', (s) => {
>   if (s.kind === 'streaming') consoleCommandWhenReady(stream, 't.MaxFPS 30');
> });
> ```
>
> The state returns to `streaming` after every reconnect, so the command runs again then. Prefer commands that are safe to repeat: `stat fps` is a toggle, `t.MaxFPS 30` is not.

> [!TIP]
>
> To change the resolution, use [`stream.setResolution()`](https://docs.streampixel.io/resources/web-sdk-v2/controls-and-input#resolution-and-quality). It needs no console-command permission.

## Launch parameters

Launch parameters are one message that the SDK sends for you when the application starts. Use them for what the application must know first: which unit to show, which language to use.

```js title="Launch parameters" highlight={4} theme={"dark"}
const stream = await Streampixel.create({
  appId: 'YOUR_PROJECT_ID',
  container: document.getElementById('stream'),
  launchParams: { unit: 'A-12', lang: 'fr' },
});
```

| Fact | Detail |
| - | - |
| How it arrives | As one UI interaction (Pixel Streaming's name for a custom message from the page to the application.), the same as `send({ unit: 'A-12', lang: 'fr' })`. Your **On Input Event** Blueprint receives it. |
| When | Shortly after the first frame, once per started application. |
| After a reconnect | Not sent again: the same application is still running. |
| After a fresh session | Sent again: a fresh session is a new application process. |
| Shared sessions | A viewer sends nothing. The host's `launchParams` are used. |
| Privacy | `launchParams` are not part of what the SDK reports to Streampixel. Your page's code is public, so do not put secrets in them. |

> [!TIP]
>
> A share link passes its URL query parameters to the application. To do the same in your page: `launchParams: Object.fromEntries(new URLSearchParams(location.search))`.

> [!NOTE]
>
> Launch parameters are a message. They are not Unreal command-line arguments. Command-line arguments are a project setting: see [Launch arguments and startup commands](https://docs.streampixel.io/resources/additional-unreal-engine-features/launch-arguments-and-startup-commands.md).

### Messages the SDK sends by itself

Besides `launchParams`, the SDK sends two things to your application on every video start. A viewer of a shared session sends neither.

| Sent | What | Switch it off |
| - | - | - |
| `{ "type": "isTouchDevice", "value": true, "os": "ios", "device": "mobile" }` | Whether the viewer uses a touch device, so your application can show touch controls. See [Hide / unhide touch controls](https://docs.streampixel.io/resources/additional-unreal-engine-features/hide-unhide-touch-controls.md). | `advanced: { flags: { isTouchDevice: false } }` |
| The project's startup console commands | The commands and delays you set in the dashboard. See [Launch arguments and startup commands](https://docs.streampixel.io/resources/additional-unreal-engine-features/launch-arguments-and-startup-commands.md). | `advanced: { flags: { startupCommands: false } }` |

## Files from the application

Your application can send a file to the viewer with the **Send File** node: a screenshot, a report, a save file. The Unreal side is on [Send files to the viewer](https://docs.streampixel.io/resources/additional-unreal-engine-features/send-files-to-the-viewer.md).

**fileProgress** `\{ received: number; total: number; name: string \}`

Fires while the file arrives. `received` and `total` count the pieces of the file, not bytes.

**file** `\{ blob: Blob; name: string; mimeType: string \}`

Fires once, when the file is complete. `name` is `download` followed by the extension your application gave, for example `download.png`: Pixel Streaming sends a file's type and extension, not its name. `mimeType` is `application/octet-stream` when the application gave none.

What happens depends on whether your page listens:

| Your page | Desktop browsers | Phones and tablets |
| - | - | - |
| No `file` listener | The file is downloaded. | Nothing happens: these browsers save a file only from a tap. |
| A `file` listener | Your listener decides. | Your listener decides. Show a Save button. |
| [`@streampixel/ui`](https://docs.streampixel.io/resources/web-sdk-v2/ui-overlays) with `files` on | Saved at once. | A Save button. |

```js title="Your own Save button" expandable theme={"dark"}
const bar = document.getElementById('file-progress'); // <progress id="file-progress" max="100" hidden></progress>
const save = document.getElementById('file-save');    // <button id="file-save" hidden>Save file</button>

stream.on('fileProgress', ({ received, total }) => {
  bar.hidden = false;
  bar.value = total ? Math.round((received / total) * 100) : 0;
});

stream.on('file', ({ blob, name }) => {
  bar.hidden = true;
  save.hidden = false;
  save.onclick = () => {                 // a tap: phones allow the download here
    const url = URL.createObjectURL(blob);
    const a = document.createElement('a');
    a.href = url;
    a.download = name || 'download';
    document.body.appendChild(a);
    a.click();
    a.remove();
    setTimeout(() => URL.revokeObjectURL(url), 60_000);
    save.hidden = true;
  };
});
```

## Links from the application

Unreal's `Launch URL` node opens a link on the server, not on the viewer's device. To open a link for the viewer, the application sends the link as the whole text of a **Send Pixel Streaming Response**.

The SDK treats a message as a link when it is one `https://` address and nothing else. A message with any other scheme is not treated as a link.

**openUrl** `\{ url: string; open: () =\> void \}`

The application sent a link. `open()` opens it in a new tab. On phones and tablets, call `open()` inside a click or tap handler: those browsers refuse a tab opened in any other way.

| Your page | Desktop browsers | Phones and tablets |
| - | - | - |
| No `openUrl` listener | The link opens in a new tab. | Nothing happens. |
| An `openUrl` listener | Your listener decides. | Your listener decides. Show a button that calls `open()`. |
| [`@streampixel/ui`](https://docs.streampixel.io/resources/web-sdk-v2/ui-overlays) with `links` on | Opened in a new tab at once. | A card with an Open button. |

```js title="A button for links" theme={"dark"}
const linkButton = document.getElementById('open-link'); // <button id="open-link" hidden></button>

stream.on('openUrl', ({ url, open }) => {
  linkButton.textContent = `Open ${new URL(url).host}`;
  linkButton.hidden = false;
  linkButton.onclick = () => { open(); linkButton.hidden = true; };
});
```

> [!NOTE]
>
> The link also reaches your `ueMessage` and response listeners, like every message. Open it in one place only.

## Screenshots

There are two ways to take one. Both capture the current video frame, without anything your page draws on top.

> [tab] Your page takes it
>
> `captureScreenshot()` returns the frame as a PNG data URL. It returns `null` before the first frame.
>
> ```js title="Screenshot button" theme={"dark"}
> document.getElementById('shot').addEventListener('click', () => {
>   const dataUrl = stream.captureScreenshot();
>   if (!dataUrl) return; // no frame yet
>   const a = document.createElement('a');
>   a.href = dataUrl;
>   a.download = 'screenshot.png';
>   document.body.appendChild(a);
>   a.click();
>   a.remove();
> });
> ```

> [tab] Your application asks for it
>
> The application sends `requestScreenshot` with **Send Pixel Streaming Response**: see [Screenshots](https://docs.streampixel.io/resources/quick-start-guide/screenshot-functionality.md). The SDK captures the frame and raises `screenshot`.
>
> ```js title="Handle the application's request" theme={"dark"}
> stream.on('screenshot', ({ dataUrl }) => {
>   document.getElementById('preview').src = dataUrl; // <img id="preview">
> });
> ```
>
> Without a `screenshot` listener, the SDK downloads the frame as `pixelstream_screenshot.png`.

For a render at a higher resolution than the stream, let the application create the image and send it as a [file](#files-from-the-application).

## Pixel Streaming events and object

Code written against Epic's library often listens to Epic's events, or uses Epic's `PixelStreaming` object directly. Both are available.

**stream.addEventListener(type, listener)** `method`

Subscribes to an event of Epic's library by its original name, for example `webRtcConnected`, `playStream` or `afkWarningActivate`. The SDK attaches your listener again whenever it rebuilds the connection. Remove it with `stream.removeEventListener(type, listener)`.

**stream.pixelStreaming** `Epic's PixelStreaming object, or null`

Epic's own object for the current connection. It is `null` before a connection exists. It is **replaced** when the SDK rebuilds the connection, so read `stream.pixelStreaming` each time you use it and never keep it in a variable.

Do not call its `reconnect()`, `unmuteMicrophone()` or `unmuteCamera()`: the SDK owns reconnection and capture.

```js title="Epic's event names on the stream object" theme={"dark"}
stream.addEventListener('playStream', () => console.log('Epic says: playing'));
```

> [!NOTE]
>
> In TypeScript, `stream.pixelStreaming` has the type `unknown`: the SDK does not export Epic's classes. Cast it to the type from Epic's package if you use that package's types.

## Troubleshooting

> [accordion] send() or emitUIInteraction() returns false
>
> The video is not up. Check `stream.state.kind`: it must be `streaming`. Use the `sendWhenStreaming` helper on this page, or send from a `state` listener.

> [accordion] It returns true and nothing happens in Unreal
>
> The message reached the stream. Check the Unreal side: the **Pixel Streaming Input** component is added, **On Input Event** is bound, and the field name in **Get Json String Value** matches the key you send. Print the descriptor in Unreal to see what arrives. The steps are on [Unreal side](https://docs.streampixel.io/resources/json-message-communication/unreal-side-sending-and-receiving-json.md).

> [accordion] consoleCommand() or emitConsoleCommand() returns false while the stream is up
>
> Either the stream started a moment ago and the application has not reported its settings yet (try again for a few seconds), or the application was launched without `-AllowPixelStreamingCommands`.

> [accordion] My listener gets a string, not an object
>
> That is how responses arrive: as the text your application sent. Parse it with `JSON.parse()` when it is JSON, as in the examples above.

> [accordion] My listener stopped after a reconnect
>
> You added it on `stream.pixelStreaming`. That object is replaced when the SDK rebuilds the connection. Add the listener on the stream object: `stream.addResponseEventListener()`, `stream.addEventListener()` or `stream.on()`.

> [accordion] A link or a file does nothing on a phone
>
> Phones and tablets open a tab or save a file only from a tap. Add an `openUrl` or `file` listener that shows a button, or mount `@streampixel/ui`.

> [accordion] The application lost its state after a pause in the video
>
> The SDK started a fresh session, which is a new application process. `launchParams` were sent again. Anything else your page sent before must be sent again: do it when the state becomes `streaming`.

## Next steps

> [card] [Controls and input](https://docs.streampixel.io/resources/web-sdk-v2/controls-and-input.md)
>
> Sound, resolution, fullscreen, keyboard, touch, microphone and camera.

> [card] [Events and states](https://docs.streampixel.io/resources/web-sdk-v2/events-and-states.md)
>
> One level up: every state and every event, with payloads.

> [card] [Cookbook](https://docs.streampixel.io/resources/web-sdk-v2/cookbook.md)
>
> Short recipes built on these calls.

> [card] [Stream API reference](https://docs.streampixel.io/resources/web-sdk-v2/stream-api.md)
>
> Every method and property of the stream object.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.