EyeCtrl

EyeCtrl is a Windows accessibility application for controlling a PC with eye tracking. It is built for the Tobii Dynavox PCEye 5 and compatible Tobii Stream Engine devices.

EyeCtrl replaces the mouse and keyboard with gaze-driven overlays. A cursor circle follows the user's gaze, dwell activates controls, an action menu provides mouse actions, and an on-screen keyboard supports text entry and prediction. All primary controls are configurable and the application supports single-monitor setups.

Features

• Gaze cursor - follows eye position with configurable adaptive smoothing, including a One Euro filter.
• Dwell activation - activates controls after a configurable gaze duration with visual progress feedback.
• Action menu - radial actions for left click, right click, double click, drag, and scrolling.
• Manual selection - magnifies a screen region for precise targeting and positions action buttons around the zoom area.
• Auto-selection - identifies WPF controls under the gaze point and exposes their actions through the dwell system.
• On-screen keyboard - QWERTY, QWERTZ, and AZERTY layouts with word prediction, emoji support, modifier keys, and optional click sounds.
• Toolbar and system tray - pause/resume, zoom, keyboard, settings, and application exit controls.
• Calibration - screen calibration with affine and polynomial transforms for physical screen coordinates.
• Eye-care support - fatigue monitoring, scheduled breaks, gaze exercises, and eye-care scoring.
• External dwell SDK - named-pipe integration for adding gaze-dwell behavior to external WPF and WinUI applications.
• Supervisor watchdog - optional crash recovery, heartbeat monitoring, fallback launching, and a supervisor tray UI.
• Diagnostics - structured Serilog logs for the application and supervisor.

Requirements

• Windows 10 or Windows 11.
• .NET 10 SDK 10.0.102, pinned in global.json.
• Tobii Dynavox PCEye 5 or another compatible Tobii eye tracker.
• Tobii Stream Engine SDK for real eye-tracker support. The build looks for Tobii.StreamEngine.Net.dll and tobii_stream_engine.dll under C:\Program Files\Tobii\Tobii EyeX\.

The application falls back to mouse-cursor-as-gaze mode when no eye tracker is connected. If the Tobii SDK DLLs are unavailable, the Platform project builds without TOBII_AVAILABLE and excludes the Tobii provider. Override the SDK location when necessary:

dotnet build EyeCtrl.slnx -p:TobiiSdkPath="C:\path\to\tobii-sdk"

Build

Build the complete solution:

dotnet build EyeCtrl.slnx

Run

Start the main application:

dotnet run --project src/EyeCtrl.App

Use test mode to skip the splash screen and force mouse-cursor-as-gaze mode:

dotnet run --project src/EyeCtrl.App -- --test-mode

The gaze ring runs the hybrid arm by default (a run-only pin with a freeze hold inside the activation button). The ring look's diagnostic arms are available per launch: --ring-arm=today or --ring-arm=run-only; every launch logs its arm (Ring arm: … [source: …]).

The supervisor is a separate Windows application. It can be launched directly when testing watchdog behavior:

dotnet run --project src/EyeCtrl.Supervisor

Publish

Create a self-contained, single-file x64 build:

dotnet publish src/EyeCtrl.App -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -o publish

Resource files such as keyboard layouts, dictionaries, emoji data, and click.wav are intentionally kept loose beside the single-file executable. Do not remove them from the published directory.

Installed (UIAccess) build

Dev, test and CI builds stay unsigned with uiAccess="false". Only an installed build runs with UIAccess, so dotnet build, dotnet test, F5 and CI never need a certificate; CI never publishes or signs. Only the owner's machine signs and installs, and only on the owner's go.

# Build, sign, catalog and deliver to %LOCALAPPDATA%\EyeCtrl\updates\<version>\
pwsh -File tools/release/Publish-Signed.ps1

# Everything except signing and the inbox copy (safe for an agent or a review)
pwsh -File tools/release/Publish-Signed.ps1 -DryRun -NoSign -ScratchDir C:\t\publish-dryrun

# Start the newest inbox package's Setup through ShellExecute (never the call operator)
pwsh -File tools/release/Install-EyeCtrl.ps1

Publish-Signed.ps1 publishes the App with -p:EyeCtrlUiAccess=true, the Supervisor (compressed single-file; the App is never compressed) and (when present) EyeCtrl.Setup into one folder, signs every exe with the pinned certificate (C1039769A2D9ED01F93CEBC78B61176EE6B12F91) and the DigiCert timestamp, verifies each signature and each embedded manifest (App uiAccess="true", Supervisor and Setup false), refuses self-extraction, then writes package.json and an EyeCtrl.cat catalog (New-FileCatalog v2) over the package and signs it. The inbox copy is assembled in a dot-prefixed .<version>.partial folder and renamed into place, so a half-written package is never visible. -DryRun implies -NoSign and refuses any destination under the update inbox, so an unsigned package can never appear there.

Test

Run all non-UI tests:

dotnet test --filter "Category!=UI"

Run the complete test suite, including tests that do not require a display:

dotnet test

Run the UI tests locally in an interactive Windows desktop session:

dotnet test tests/EyeCtrl.App.Tests --filter "Category=UI"

The test suite uses xUnit v3 and NFluent. UI tests use FlaUI and are not run in headless CI. GitHub Actions builds the solution, validates the theme palettes, runs the non-UI tests with a coverage-threshold gate, and runs the benchmark generator tests on pushes to main and pull requests targeting main.

Benchmarks

Benchmarks live in benchmarks/EyeCtrl.Benchmarks/ and use BenchmarkDotNet. The benchmark project is standalone and is not included in EyeCtrl.slnx.

dotnet run --project benchmarks/EyeCtrl.Benchmarks -c Release -- --filter * --job short
dotnet run --project benchmarks/EyeCtrl.Benchmarks -c Release -- --filter *Bigram*
dotnet run --project benchmarks/EyeCtrl.Benchmarks -c Release -- --filter *Gaze*
dotnet run --project benchmarks/EyeCtrl.Benchmarks -c Release -- --filter *WithEyeCare*
dotnet run --project benchmarks/EyeCtrl.Benchmarks -c Release -- --filter *SmallLM*

SmallLM benchmarks require an ONNX model and tokenizer under %APPDATA%\EyeCtrl\models. Set EYECTRL_MODEL_DIR to use another directory. If no model is available, the SmallLM benchmark class is skipped.

Project Structure

The solution uses a layered architecture with separate SDK, eye-care, and supervisor components:

src/
  EyeCtrl.Core/              Framework-agnostic gaze, dwell, settings, and prediction logic
  EyeCtrl.Platform/          Win32 input, mouse fallback, and conditional Tobii integration
  EyeCtrl.Controls/          WPF control discovery for auto-selection mode
  EyeCtrl.ViewModels/        MVVM view models and application messages
  EyeCtrl.App/               WPF application, overlays, settings, tray, and composition root
  EyeCtrl.LM/                ONNX language model and tokenizer
  EyeCtrl.EyeCare/           Fatigue monitoring, breaks, exercises, and scoring
  EyeCtrl.Contracts/         Named-pipe wire messages shared by the SDK and app
  EyeCtrl.Sdk.Abstractions/  Framework-agnostic external dwell client and adapters
  EyeCtrl.Sdk.Wpf/           WPF attached properties for external dwell
  EyeCtrl.Sdk.WinUI/         WinUI 3 attached properties for external dwell
  EyeCtrl.Supervisor/        Watchdog, crash recovery, fallback, and supervisor UI
tests/
  EyeCtrl.*.Tests/           Unit, integration, SDK, eye-care, supervisor, and UI tests
benchmarks/
  EyeCtrl.Benchmarks/        Standalone BenchmarkDotNet project
docs/
  plans/                     Design documents and implementation plans
  superpowers/               Superpowers design specs and implementation plans

The main application follows App -> ViewModels/Platform -> Core. The SDK layer is independent of the application internals and communicates with EyeCtrl.App through named-pipe IPC. The supervisor is a separate process that monitors the app through heartbeat messages and can relaunch it or start a configured fallback application.

See ARCHITECTURE.md for the dependency graph, execution flows, functional areas, and SDK protocol details.

External Dwell SDK

External applications can opt controls into gaze dwell without depending on the WPF app internals:

• EyeCtrl.Contracts defines the named-pipe wire format.
• EyeCtrl.Sdk.Abstractions provides DwellClient and framework-neutral element adapters.
• EyeCtrl.Sdk.Wpf provides WPF attached properties such as Dwell.Enabled and Dwell.Mode.
• EyeCtrl.Sdk.WinUI provides the corresponding WinUI 3 integration.

The running EyeCtrl app hosts the named-pipe server and renders dwell feedback for registered external controls.

Troubleshooting

• No eye tracker connected: EyeCtrl starts in mouse-cursor-as-gaze fallback mode.
• Tobii provider unavailable: install the Tobii Stream Engine SDK or build with -p:TobiiSdkPath="...".
• Application starts paused: check the StartPaused setting or whether it was launched with the supervisor's --paused retry handoff.
• Single-file publish is missing resources: keep JSON layouts, dictionaries, emoji data, and click.wav beside the executable.
• Runtime diagnostics: application logs are written to %APPDATA%\EyeCtrl\logs\eyectrl-.log with daily rolling and seven-day retention. Supervisor logs use %APPDATA%\EyeCtrl\logs\supervisor-.log.
• UI tests do not run: run them in an interactive Windows desktop session with dotnet test --filter "Category=UI".

Development

• Use file-scoped namespaces, Allman braces, and _camelCase fields.
• Keep Core free of native ML dependencies; use ILanguageModelPredictor for the language-model seam.
• Tests use private nested fakes rather than a mocking library.
• Run tools/gen_dicts.py when keyboard word-frequency dictionaries need updating.
• No linter or formatter is configured; dotnet build and dotnet test are the primary verification commands, and CI additionally gates on the theme palettes and coverage thresholds.

See AGENTS.md for detailed build commands, testing conventions, dependency quirks, and repository guidance.

Key Files

File	Purpose
src/EyeCtrl.App/App.xaml.cs	Application entry point, overlay lifecycle, and tray icon
src/EyeCtrl.App/Services/AppBootstrapper.cs	Dependency-injection composition root
src/EyeCtrl.App/Services/EventCoordinator.cs	Dwell, zoom, keyboard, and action-menu coordination
src/EyeCtrl.Core/GazeEngine.cs	Core gaze-processing pipeline
src/EyeCtrl.Core/Settings/SettingsService.cs	JSON settings persistence and validation
src/EyeCtrl.Supervisor/Program.cs	Supervisor startup and watchdog orchestration
src/EyeCtrl.Supervisor/Supervision/SupervisionStateMachine.cs	Crash-recovery state transitions
src/EyeCtrl.Contracts/	External SDK named-pipe contracts

License

Private project. Not licensed for redistribution.