> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streampixel.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Controls and input

> Control sound, autoplay, resolution, fullscreen, mouse, keyboard, touch, gamepad, XR, text input on phones, and the viewer's microphone and camera with the Streampixel Web SDK.

Intermediate For web developers who build their own buttons around a stream. After this page you can add a sound button, a resolution picker, fullscreen, text input for phones, and microphone and camera switches.

> [!TIP]
>
> You do not have to build any of this. [`@streampixel/ui`](https://docs.streampixel.io/resources/web-sdk-v2/ui-overlays) draws a control bar with sound, stream information and fullscreen, the "Click to start" screen and the text field for phones. Read on when you want your own.

Every setting on this page has the same three layers:

| Layer | Where | Wins when |
| - | - | - |
| The project's setting | The Streampixel dashboard | Your code says nothing. |
| An option | `Streampixel.create({ ... })` | You pass it. It applies from the start of the stream. |
| A method | The `stream` object | You call it while the stream is up. |

For the settings that cost GPU time or bandwidth, the dashboard is the upper limit: your code can ask for less, never more. The full list is in [Options](https://docs.streampixel.io/resources/web-sdk-v2/options.md).

## Sound and autoplay

**A stream starts muted.** Browsers only allow sound after the viewer has clicked or tapped. Give the viewer a sound button.

**1. Add a button**

```html theme={"dark"}
<button id="sound" hidden>Unmute</button>
```

**2. Switch the sound in its click handler**

```js title="Sound button" theme={"dark"}
const soundButton = document.getElementById('sound');

soundButton.addEventListener('click', () => {
  const soundOn = stream.toggleAudio();   // true: the sound is now on
  soundButton.textContent = soundOn ? 'Mute' : 'Unmute';
});

stream.on('state', (s) => {
  if (s.kind !== 'streaming') return;
  soundButton.hidden = false;
  soundButton.textContent = stream.muted ? 'Unmute' : 'Mute';
});
```

> [!TIP]
>
> The video plays without sound. After one click on the button you hear the application.

**stream.muted** `boolean`

Whether the sound is muted now. `true` before the video exists.

**stream.setMuted(muted)** `boolean`

Mutes or unmutes. Returns the muted state now in effect. Call it from a click or tap when you unmute.

**stream.toggleAudio()** `boolean`

Flips mute. Returns `true` when the sound is now **on**. Note the direction: it is the opposite of `muted`.

> [!NOTE]
>
> Read `stream.muted` each time the state becomes `streaming`, as the example does. When the SDK rebuilds a connection it creates a new video element, and a new video element starts muted.

### When the browser blocks autoplay

Some browsers refuse to start even a muted video without a click. The stream is then up, and nothing plays.

| Signal | Value |
| - | - |
| State | `awaitingPlay`, with `reason` |
| Event | `autoplayBlocked`, with `{ reason }` |

Show a "Click to start" control that calls `stream.play()`:

```js title="Click to start" theme={"dark"}
const startButton = document.getElementById('start'); // <button id="start" hidden>Click to start</button>

stream.on('state', (s) => {
  startButton.hidden = s.kind !== 'awaitingPlay';
});
startButton.addEventListener('click', () => stream.play());
```

When you passed a `container`, a press anywhere on it also starts playback. The state becomes `streaming` when the video really plays.

## Resolution and quality

### How the video fits its container

The resolution mode decides the size of the stream and how it sits in your container. It is the project's dashboard setting, or the `resolution.mode` option.

| Mode | Stream size | In the container |
| - | - | - |
| `fixed` | The start resolution, or the size last set with `setResolution()`. | The whole picture is visible, with bars where the shapes differ. |
| `dynamic` | The container's size in device pixels (The size in CSS pixels multiplied by the screen's devicePixelRatio.), never above the start resolution (or the size set with `setResolution()`). Follows the container when it or the window changes size. | Fills the container. |
| `crop` | The same as `dynamic`. | Fills the container edge to edge. What does not fit is cut off. |

> [!WARNING]
>
> For `dynamic` and `crop`, give the container a width **and** a height. A container with no size is measured as the whole browser window.

### Change the resolution

**1. List the sizes the project allows**

`stream.allowedResolutions` holds the sizes up to the project's maximum, for example `"720p (1280x720)"`.

**2. Ask for one**

`stream.setResolution(width, height)` asks the application to render at that size.

**3. Read what was applied**

It returns the size that was sent, or `null` when nothing was sent.

```js title="Resolution picker" expandable theme={"dark"}
const picker = document.getElementById('resolution'); // <select id="resolution"></select>

stream.on('state', (s) => {
  if (s.kind !== 'streaming' || picker.options.length) return;
  for (const label of stream.allowedResolutions) {
    picker.add(new Option(label.split(' ')[0], label));   // "1080p", value "1080p (1920x1080)"
  }
});

picker.addEventListener('change', () => {
  const match = picker.value.match(/\((\d+)x(\d+)\)/);
  if (!match) return;
  const applied = stream.setResolution(Number(match[1]), Number(match[2]));
  console.log(applied ? `Now ${applied.width}x${applied.height}` : 'Not applied');
});
```

What `setResolution()` does with your numbers:

| Rule | Detail |
| - | - |
| Never above the maximum | The size is lowered to the maximum, keeping its shape. The maximum is the project's maximum stream quality, or a lower `resolution.max` that you passed. |
| Even numbers | Both values are rounded to even numbers. |
| Too small or not a number | Values under 64, and anything that is not a number, are ignored: `null`. |
| Before the stream is up | Returns `null`. The size is remembered and applied when the stream starts. |
| After a reconnect | Kept. The stream comes back at the size you last asked for. |
| `dynamic` and `crop` | The size becomes the new upper limit, and the stream is fitted to the container under it. |
| Shared sessions | Always `null` for a viewer: the size is the host's. |
| Permission | None needed. It does not use a console command. |

### Set quality at the start

These options apply from the first frame. Each defaults to the project's dashboard setting.

```js title="Quality options" highlight={4-6} theme={"dark"}
const stream = await Streampixel.create({
  appId: 'YOUR_PROJECT_ID',
  container: document.getElementById('stream'),
  resolution: { start: '1080p (1920x1080)', mobileStart: '720p (1280x720)', mode: 'dynamic' },
  codec: { primary: 'AV1', fallback: 'H264' },
  bitrate: { min: 1_000, max: 20_000 },   // kilobits per second: 1 to 20 Mbps
});
```

The SDK orders the codecs that the viewer's browser can decode. A browser without the primary codec uses the fallback, then its own default. AV1 is used only when the application was built with Unreal Engine above 5.3.

> [!WARNING]
>
> `bitrate` is in **kilobits per second**. `20_000` is 20 Mbps. A value above 200,000 logs a console warning, because it looks like bits per second.

To show the viewer what they are getting, read the `stats` event: frame rate, bitrate, resolution, codec. See [`StreamStats`](https://docs.streampixel.io/resources/web-sdk-v2/events-and-states#streamstats) and the [stats readout recipe](https://docs.streampixel.io/resources/web-sdk-v2/cookbook.md#a-stats-readout).

## Fullscreen

The SDK has no fullscreen method. Use the browser's own Fullscreen API on the element that holds the stream.

```html title="index.html" theme={"dark"}
<div id="player" style="position: relative; width: 100%; aspect-ratio: 16 / 9;">
  <div id="stream" style="width: 100%; height: 100%;"></div>
  <button id="fullscreen">Fullscreen</button>
</div>
```

```js title="Fullscreen button" theme={"dark"}
const player = document.getElementById('player');
const fullscreenButton = document.getElementById('fullscreen');

// Some browsers do not offer fullscreen for page elements.
fullscreenButton.hidden = typeof player.requestFullscreen !== 'function';

fullscreenButton.addEventListener('click', () => {
  if (document.fullscreenElement) document.exitFullscreen();
  else player.requestFullscreen();
});
```

* Make the **wrapper** fullscreen, not the stream container: your own buttons inside the wrapper then stay visible.
* In `dynamic` and `crop` modes the stream follows the container, so it is rendered at the new size after a moment. In `fixed` mode the picture is scaled.

## Mouse, keyboard, touch and gamepad

Input goes to your application as soon as the video is up. Each kind follows the project's dashboard setting ([Input and UI controls](https://docs.streampixel.io/resources/quick-start-guide/frontend-settings.md)) unless you pass an option.

```js title="Input options" highlight={4} theme={"dark"}
const stream = await Streampixel.create({
  appId: 'YOUR_PROJECT_ID',
  container: document.getElementById('stream'),
  input: { keyboard: false, gamepad: true },   // everything not named follows the dashboard
});
```

**input.mouse** `boolean` (default: dashboard, else on)

Mouse movement, buttons and wheel.

**input.keyboard** `boolean` (default: dashboard, else on)

Keys.

**input.touch** `boolean` (default: dashboard, else on)

Touches on phones, tablets and touch screens.

**input.fakeMouseWithTouches** `boolean` (default: dashboard, else off)

Sends touches as mouse events. For an application that has no touch handling of its own.

**input.gamepad** `boolean` (default: dashboard, else off)

Gamepads connected to the viewer's device.

**input.hover** `boolean` (default: dashboard, else off)

Hover mouse. See [below](#hover-mouse).

**input.xr** `boolean` (default: dashboard, else off)

WebXR controllers. See [VR and XR](#vr-and-xr).

> [accordion] What the SDK does with input for you
>
> | Situation | What the SDK does |
> | - | - |
> | The viewer types in your page's own `<input>`, `<textarea>`, `<select>` or editable element | Those keys are **not** sent to the application. Your search box and chat box work normally. |
> | The tab is hidden or the window loses focus | Keys and mouse buttons that were held are released in the application, so a character does not keep walking. |
> | The viewer's first mouse press on the stream | The SDK first sends one middle-button press and release at position 0,0, then the viewer's press. An application that runs off-screen spends the first press it receives on activating its window; this way the viewer's first click counts. Your application sees that extra press once per video start. Touch and pen presses are left alone. |
> | The viewer is on a touch device | The application is told with an `isTouchDevice` message. See [Talk to your Unreal app](https://docs.streampixel.io/resources/web-sdk-v2/messaging.md#messages-the-sdk-sends-by-itself). |

> [accordion] Shared sessions
>
> A viewer of a [shared session](https://docs.streampixel.io/resources/web-sdk-v2/shared-sessions.md) watches only. Mouse, keyboard, touch, gamepad, microphone and camera are off for a viewer, whatever the options say. The host has the input.

### Hover mouse

With hover mouse on, mouse movement reaches the application without a pressed button. Whether your application wants that depends on how it shows its cursor: see [Mouse settings](https://docs.streampixel.io/resources/quick-start-guide/mouse-settings.md).

Three things can switch it:

| Who | How |
| - | - |
| The dashboard or your options | The **Hovering mouse** setting, or `input: { hover: true }`. |
| Your page, while streaming | `stream.setHoverMouse(true)` or `stream.setHoverMouse(false)`. |
| Your application, while streaming | The message `{ "type": "togglehoveringmouse", "value": true }`. The SDK acts on it by itself. |

> [!NOTE]
>
> A change made while streaming belongs to the current connection. When the SDK rebuilds the connection, hover mouse starts again from the option or the dashboard. If your page changed it, set it again when the state returns to `streaming`.

### VR and XR

| Call or option | What it does |
| - | - |
| `input: { xr: true }` | Sends WebXR controller input to the application. |
| `stream.toggleXR()` | Enters or leaves the immersive session. Call it from a click or tap. It does nothing on a device without VR support. |
| `advanced: { flags: { autoEnterVR: false } }` | By default a headset browser enters VR by itself. This switches that off. |

The Unreal side is on [Prepare your Unreal Engine project for VR](https://docs.streampixel.io/resources/quick-start-guide/prepare-your-unreal-engine-project-for-vr.md).

## Text input on phones

A phone has no keyboard until a text field on the page has focus. When a text field in your application gets focus, the application asks the page for one. The Unreal side is on [Virtual keyboard for text input](https://docs.streampixel.io/resources/additional-unreal-engine-features/virtual-keyboard.md).

**1. Add a text field to your page**

```html theme={"dark"}
<form id="osk" hidden>
  <input id="osk-text" autocomplete="off" />
  <button>Send</button>
</form>
```

**2. Show it on osk, answer with sendTextboxEntry()**

```js title="Text input for the application" theme={"dark"}
const form = document.getElementById('osk');
const field = document.getElementById('osk-text');

stream.on('osk', ({ contents }) => {   // contents: the text already in the application's field
  field.value = contents;
  form.hidden = false;
  field.focus();
});

stream.on('oskHide', () => { form.hidden = true; });

form.addEventListener('submit', (e) => {
  e.preventDefault();
  stream.sendTextboxEntry(field.value);
  form.hidden = true;
});
```

> [!TIP]
>
> Tap a text field in the stream on a phone. Your field appears with the same text. Type, press Send, and the text is in the application's field.

* Keys typed into your field are not sent to the application as key presses. Only `sendTextboxEntry()` sends the text.
* Some phones open the keyboard only when the viewer taps your field. Place the field where it is easy to tap.
* The event fires on every device, not only on phones.

## Microphone and camera

The viewer's microphone or camera can be sent **to your application**, for example for voice commands. This is not voice chat between people: for that, see [Voice and text chat](https://docs.streampixel.io/resources/web-sdk-v2/voice-chat.md).

**1. Allow it at the start**

Switch it on in the dashboard, or pass `media`. This must be decided before the stream starts.

```js title="Allow the microphone" highlight={4} theme={"dark"}
const stream = await Streampixel.create({
  appId: 'YOUR_PROJECT_ID',
  container: document.getElementById('stream'),
  media: { mic: true },
});
```

With the microphone allowed, the SDK asks the browser for it as soon as the video is up, and sends it once the viewer agrees. A prompt that the viewer ignores never delays the video.

**2. Switch it while streaming**

```js title="Microphone button" theme={"dark"}
const micButton = document.getElementById('mic'); // <button id="mic">Turn microphone off</button>
let micOn = true;   // the SDK asked for the microphone when the video started

micButton.addEventListener('click', async () => {
  micOn = await stream.setMicrophone(!micOn);   // resolves to whether the microphone is live now
  micButton.textContent = micOn ? 'Turn microphone off' : 'Turn microphone on';
});
```

The SDK does not report the viewer's answer to the first permission prompt. `setMicrophone(true)` does: it resolves to `true` only when the microphone is live.

**stream.setMicrophone(on)** `Promise\<boolean\>`

Turns the viewer's microphone on or off while the stream is up. Resolves to whether it is live now. Call it from a click or tap: the first call asks the browser for permission.

**stream.setCamera(on)** `Promise\<boolean\>`

The same for the camera (`media.camera`).

Both resolve to `false` in these cases:

| Case | Why |
| - | - |
| The microphone or camera was not allowed at the start (`media`, or the dashboard) | The connection was set up without it. Nothing is captured. |
| The viewer refused the browser's permission prompt | The browser gives no access. |
| The stream is not up | There is no connection to attach it to. |
| A viewer of a shared session | Viewers send nothing. |

> [!NOTE]
>
> The microphone is sent as it is: one channel, with the browser's echo cancellation, noise suppression and automatic gain switched off.

## Idle viewers and background tabs

Input also tells the platform that the viewer is still there.

| Situation | What happens |
| - | - |
| No input for the idle time (the project's setting, or `afk.timeoutSec`, which can only make it shorter) | The `afkWarning` event counts down, then the session ends with `endKind: 'afk'`. |
| The viewer is watching, not clicking (a video, a cutscene) | Call `stream.keepAlive()`. It restarts the idle timer and cancels a countdown. |
| A phone or tablet leaves the tab in the background for 60 seconds | The session ends. Switch this off with `advanced: { flags: { backgroundDisconnect: false } }`. |

The recipe for your own warning is in the [cookbook](https://docs.streampixel.io/resources/web-sdk-v2/cookbook.md#an-idle-warning).

## Troubleshooting

> [accordion] There is no sound
>
> Streams start muted. Call `stream.setMuted(false)` or `stream.toggleAudio()` from a click or tap. Outside a click the browser may refuse.

> [accordion] The sound went away after a reconnect
>
> The SDK built a new connection, with a new video element that starts muted. Read `stream.muted` when the state becomes `streaming` and update your button.

> [accordion] The video is a black box until I click
>
> The browser blocked autoplay. The state is `awaitingPlay`. Show a button that calls `stream.play()`.

> [accordion] setResolution() returns null
>
> The stream is not up yet, a value is under 64, or this is a viewer of a shared session.

> [accordion] The stream has the wrong size for its container
>
> In `dynamic` and `crop` modes the stream is sized to the container. Check that the container has a width and a height. Without them the SDK measures the browser window.

> [accordion] Keys I type in my own form reach the application
>
> Keys typed in `<input>`, `<textarea>`, `<select>` and `contenteditable` elements are held back. A custom widget built from other elements is not recognised.

> [accordion] setMicrophone(true) resolves to false
>
> The microphone was not allowed at the start. Pass `media: { mic: true }` to `create()` or switch it on in the dashboard, then start a new stream.

> [accordion] The first click does something unexpected in my application
>
> Before the viewer's first mouse press the SDK sends one middle-button press and release at position 0,0. If your application reacts to the middle button, ignore a press at that position.

## Next steps

> [card] [Talk to your Unreal app](https://docs.streampixel.io/resources/web-sdk-v2/messaging.md)
>
> Messages, console commands, files and screenshots.

> [card] [Events and states](https://docs.streampixel.io/resources/web-sdk-v2/events-and-states.md)
>
> One level up: every state and event behind these controls.

> [card] [Options reference](https://docs.streampixel.io/resources/web-sdk-v2/options.md)
>
> Every option of `create()`, with types and defaults.

> [card] [Stream API reference](https://docs.streampixel.io/resources/web-sdk-v2/stream-api.md)
>
> Every method and property of the stream object.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.