neko-web-ui-research.md

Neko Web UI Research - kernel-images Repository

Overview

This document explains how the kernel-images repository uses neko to expose a web UI for the browser running in the container at port 8080.

Architecture

The kernel-images repository uses neko as a WebRTC-based remote desktop solution with the following architecture:

1. Neko Server: Runs the neko server binary (/usr/bin/neko) on port 8080
2. WebRTC Streaming: Uses WebRTC for real-time video/audio streaming of the desktop
3. Vue.js Client: Includes a custom Vue.js client built from source
4. X11 Desktop: Streams an X11 desktop running Chromium browser
5. Fallback Option: Uses noVNC as fallback when WebRTC is disabled

Client-Side Code Location

Source Files

• Location: /home/mike/src/kernel-images/images/chromium-headful/client/
• Framework: Vue.js 2 application with TypeScript

Key Files to Study

1. Entry Point

   • client/src/main.ts - Mounts the Vue app to #neko element
   • Sets up plugins (Logger, Notifications, ToolTip, Axios, etc.)
   • Initializes the neko client

2. Main Components

   • client/src/app.vue - Main application component
   • client/src/components/video.vue - Main video streaming component that handles WebRTC
   • client/src/components/controls.vue - Control interface
   • client/src/components/connect.vue - Connection interface

3. WebRTC Implementation

   • client/src/neko/index.ts - WebSocket and WebRTC client implementation
   • client/src/neko/base.ts - Base client class with WebRTC peer connection logic
   • client/src/neko/messages.ts - Message types for WebSocket communication
   • client/src/neko/events.ts - Event definitions

4. HTML Template

   • client/public/index.html - The HTML template served at port 8080

Build Configuration

• client/package.json - Dependencies and build scripts
• client/vue.config.js - Vue CLI configuration
• Build script: npm run build (also has build:lib for library build)

Customizations

The client has several features disabled (marked with "KERNEL" comments):

• Fullscreen button disabled
• Resolution controls disabled
• Clipboard controls disabled
• Picture-in-Picture disabled
• Play/unmute overlays disabled

Server-Side Configuration

Docker Setup

From the Dockerfile:

# Copy neko binary from official image
COPY --from=neko /usr/bin/neko /usr/bin/neko

# Copy built client files to web root
COPY --from=client /src/dist/ /var/www

# Copy configuration
COPY neko.yaml /etc/neko/neko.yaml

Neko Configuration (neko.yaml)

desktop:
  screen: "1920x1080@60"

member:
  provider: "noauth"

session:
  merciful_reconnect: true
  implicit_hosting: true
  cookie:
    enabled: false  # disabled for legacy API

plugins:
  enabled: false

chat:
  enabled: false

filetransfer:
  enabled: false

Server Startup

From wrapper.sh:

if [[ "${ENABLE_WEBRTC:-}" == "true" ]]; then
  # use webrtc
  echo "✨ Starting neko (webrtc server)."
  /usr/bin/neko serve --server.static /var/www --server.bind 0.0.0.0:8080 >&2 &
else
  # use novnc
  ./novnc_startup.sh
  echo "✨ noVNC demo is ready to use!"
fi

Key parameters:

• --server.static /var/www - Serves static client files
• --server.bind 0.0.0.0:8080 - Binds to all interfaces on port 8080

How It Works

1. Initial Page Load

   • User visits http://localhost:8080
   • Neko server serves the static index.html from /var/www
   • HTML loads the Vue.js application

2. Connection Establishment

   • Vue app initializes and creates WebSocket connection to the server
   • WebSocket URL is constructed based on current location
   • Authentication happens via URL parameters (username/password)

3. WebRTC Negotiation

   • WebRTC offer/answer exchange happens over WebSocket
   • ICE candidates are exchanged for NAT traversal
   • Peer connection is established

4. Streaming

   • Video stream from X11 desktop is captured
   • Audio is captured from PulseAudio
   • Streams are sent via WebRTC to the client
   • Client displays video in <video> element

5. User Interaction

   • Mouse/keyboard events are captured in overlay textarea
   • Events are sent via WebRTC data channel
   • Server injects events into X11 using custom neko input driver

Integration Options for React Apps

Option 1: iframe Embedding

// Simple iframe embedding
<iframe 
  src="http://server:8080/?usr=neko&pwd=neko&embed=1"
  allowFullScreen
  webkitallowfullscreen="true"
  mozallowfullscreen="true"
  allow="fullscreen *"
/>

Benefits:

• No code changes needed
• embed=1 parameter hides sidebar/top bar
• Complete isolation from parent app

Option 2: Library Build

The client supports building as a library:

npm run build:lib

This creates a library build that exports Vue components:

• NekoVideo
• NekoControls
• NekoConnect
• etc.

Current limitation: Vue 2 based, would need wrapper for React

Option 3: Custom WebRTC Client

Build a custom React client that:

1. Connects to neko WebSocket API
2. Handles WebRTC negotiation
3. Displays video stream
4. Sends input events

This requires understanding the neko protocol and significant development effort.

Future Considerations

According to the neko roadmap, V3 will include:

• A modular TypeScript client library without Vue dependency
• Better extensibility for embedding in other applications
• Improved component architecture

This would make React integration much easier in the future.

Additional Notes

• The repository includes custom X11 input drivers for mouse/keyboard injection
• Chrome is run with specific flags for better compatibility
• The setup includes cleanup of Chrome lock files to prevent startup issues
• D-Bus is configured to reduce error messages
• Scale-to-zero functionality is managed for Unikraft Cloud deployment