# Communicating with Unreal Engine

The SDK enables bidirectional communication between your web application and Unreal Engine.

## Video Tutorials

{% embed url="<https://youtube.com/watch?v=GgUBZ24aRKc>" %}
Send JSON from Frontend to Unreal Engine on Launch
{% endembed %}

{% embed url="<https://youtube.com/watch?v=cKGg9q7Iazs>" %}
Launch URL and Open Websites from Pixel Streaming
{% endembed %}

## Sending Data to UE

### UI Interaction (Custom JSON)

Send arbitrary JSON data to UE. This is the primary method for custom web-to-UE communication.

{% code lineNumbers="true" %}

```javascript
appStream.stream.emitUIInteraction({
  type: 'action',
  value: 'jump'
});

appStream.stream.emitUIInteraction({
  command: 'setColor',
  color: '#ff0000',
  intensity: 0.8
});
```

{% endcode %}

In Unreal Engine, handle these messages using the **Pixel Streaming Input Component's** `OnInputEvent` delegate or the `FPixelStreamingInputHandler` in C++.

### Console Commands

Execute Unreal Engine console commands remotely:

```javascript
// Show FPS counter in UE
pixelStreaming.emitConsoleCommand('stat fps');

// Set resolution
pixelStreaming.emitConsoleCommand('r.SetRes 1920x1080f');

// Other UE console commands
pixelStreaming.emitConsoleCommand('stat unit');
pixelStreaming.emitConsoleCommand('r.ScreenPercentage 100');
```

{% hint style="warning" %}
Console commands execute with the permissions of the UE application. Only commands that are enabled and allowed in your UE build will work. Ensure your application properly restricts which commands can be executed remotely.
{% endhint %}

### Text Input

Send text to UE's currently focused text field:

```javascript
pixelStreaming.streamMessageController
  .toStreamerHandlers.get('TextboxEntry')?.(['Hello from the web!']);
```

This is primarily used in response to UE's on-screen keyboard request (see below).

## Receiving Data from UE

### Response Event Listener

Listen for custom messages sent from UE to the web application:

{% code lineNumbers="true" %}

```javascript
pixelStreaming.addResponseEventListener('handle_responses', (response) => {
  const data = JSON.parse(response);
  console.log('Received from UE:', data);

  // Handle different message types
  switch (data.type) {
    case 'gameState':
      updateGameUI(data.state);
      break;
    case 'notification':
      showNotification(data.message);
      break;
  }
});
```

{% endcode %}

In Unreal Engine, send messages using the `UPixelStreamingDelegates::SendPixelStreamingResponse` function or the equivalent Blueprint node.

## On-Screen Keyboard

UE can request text input from the user by sending a `showOnScreenKeyboard` command. The SDK handles this automatically:

1. UE sends `showOnScreenKeyboard` with the current text content and cursor position
2. The SDK displays a floating text input modal (EditTextModal)
3. The user types and confirms
4. The SDK sends the text back to UE via the `TextboxEntry` handler

{% hint style="info" %}
The on-screen keyboard flow is fully automatic. No additional code is needed on the web side — the SDK handles displaying the input modal and sending the text back to UE.
{% endhint %}

## Bidirectional Communication Example

```javascript
// Web → UE: Request current score
appStream.stream.emitUIInteraction({
  type: 'request',
  action: 'getScore'
});

// UE → Web: Receive the score
pixelStreaming.addResponseEventListener('handle_responses', (response) => {
  const data = JSON.parse(response);
  if (data.type === 'scoreUpdate') {
    document.getElementById('score').textContent = data.score;
  }
});
```