Proposal: manifest-defined, permissioned app actions for custom Codex pets

What variant of Codex are you using?

Codex App (Desktop), Microsoft Store/AppX package 26.715.4045.0, on Windows 11 x64 (build 26200), with a locally installed custom V2 pet.

What feature would you like to see?

Summary

Add a small, additive pet.json contract that lets a custom Codex pet bind gestures to actions already advertised by Codex or an installed/authorized app.

This is an implementation-focused child proposal for #21657, which requests broader custom-pet interaction hooks and APIs. It is complementary to #20863, which requests manifest-defined animation behavior.

The narrow first version proposed here is:

A pet may reference an existing action ID. It may not define executable code, request new connector scopes, store credentials, or bypass the existing approval system.

Concrete use case

I created and installed a valid custom V2 pet named Companion:

• 1536 ? 1872 WebP atlas
• 8 ? 9 grid
• RGBA transparency
• all required animation rows
• validator completed with zero errors and zero warnings

The pet works as an animated status surface, but the runtime owns every interaction. Clicking it performs the built-in Codex behavior; there is no supported way for pet.json to bind a gesture to an approved app action.

A concrete example would be music control:

• single click: play/pause
• double click: next track
• context-menu action: open the music app
• optional pet animation while playback is active

Apple Music is only an example. This proposal does not assume that the current Apple Music integration already exposes playback actions. A binding should become available only when an installed app explicitly advertises the corresponding action.

What this would enable

The same contract could support:

• play/pause or next-track controls for a music app;
• starting an approved timer;
• opening a project-specific Codex thread or dashboard;
• invoking a safe, named Codex action;
• showing a context menu of approved workflow actions;
• accessibility switches or hardware buttons using the same action registry;
• reusing the native floating pet instead of building a second desktop overlay.

Current behavior / reproducible gap

1. Install a valid V2 custom pet under the supported custom-pet directory.
2. Refresh custom pets and select it.
3. Wake the pet.
4. Connect an app that advertises one or more safe actions, if available.
5. Click or double-click the custom pet.
6. Observe that only the built-in pet interaction runs.
7. Add an interactions field to pet.json.
8. Observe that there is currently no documented schema, validator, dispatcher, permission prompt, or runtime API for it.

This is not an asset or atlas problem. The missing surface is the supported interaction contract.

Proposed additive manifest

Illustrative schema:

{
  "id": "companion",
  "displayName": "Companion",
  "description": "A custom Codex pet with optional approved actions.",
  "spritesheetPath": "spritesheet.webp",
  "interactionVersion": 1,
  "interactions": {
    "click": {
      "type": "appAction",
      "appId": "apple-music",
      "actionId": "music.playback.toggle"
    },
    "doubleClick": {
      "type": "appAction",
      "appId": "apple-music",
      "actionId": "music.playback.next"
    },
    "contextMenu": [
      {
        "label": "Open music app",
        "action": {
          "type": "appAction",
          "appId": "apple-music",
          "actionId": "app.open"
        }
      }
    ]
  }
}

The identifiers above are illustrative, not claims about current Apple Music action names.

Existing pets without interactionVersion or interactions must retain current behavior.

Suggested TypeScript contract

type PetGesture =
  | "click"
  | "doubleClick"
  | "contextMenu"
  | "keyboardShortcut";

type PetActionBinding =
  | {
      type: "codexAction";
      actionId: string;
    }
  | {
      type: "appAction";
      appId: string;
      actionId: string;
    };

interface PetInteractionManifestV1 {
  interactionVersion: 1;
  interactions?: Partial<
    Record<Exclude<PetGesture, "contextMenu">, PetActionBinding>
  > & {
    contextMenu?: Array<{
      label: string;
      action: PetActionBinding;
    }>;
  };
}

A V1 binding intentionally contains no shell command, executable path, OAuth token, arbitrary URL, or manifest-supplied payload.

Suggested dispatcher boundary

async function handlePetGesture(
  pet: InstalledPet,
  gesture: PetGesture,
): Promise<void> {
  const binding = resolvePetBinding(pet, gesture);

  if (!binding) {
    return runDefaultPetInteraction(gesture);
  }

  const action = actionRegistry.resolve(binding);

  if (!action || !action.advertised) {
    return showPetActionUnavailable(binding);
  }

  const decision = await permissionBroker.authorize({
    source: {
      type: "customPet",
      id: pet.id,
      displayName: pet.displayName,
    },
    action: action.descriptor,
  });

  if (!decision.allowed) {
    return runDefaultPetInteraction(gesture);
  }

  await action.invoke();
}

The existing app/connector action registry and approval broker should remain authoritative. The pet is only another user-interface entry point.

Security and permission requirements

• Installing a pet must not install an app or grant connector scopes.
• Only actions advertised by an installed and authorized app may resolve.
• Existing action side-effect annotations and approval policy must still apply.
• No arbitrary commands, executable paths, dynamic code, credentials, or raw OAuth tokens in pet.json.
• Prefer no manifest-supplied action arguments in V1.
• Attribute every approval prompt to the pet and target app.
• Rate-limit and debounce gesture actions.
• A missing, unavailable, or denied action must safely fall back to the existing pet behavior.
• Managed workspaces must be able to disable pet action bindings independently of ordinary pet display.
• Pet packages remain local and should not receive ambient access to conversation content.

Interaction conflicts to resolve

This proposal should compose with:

• #32377: Stay and Draggable modes
• #32776: click-to-toggle the Codex window
• #20863: configurable animation and activity events

A reasonable event priority would be:

1. pointer-down/drag threshold handles repositioning;
2. context-menu gesture opens the pet menu;
3. configured double-click wins over single-click after a short gesture window;
4. unbound or denied gestures use the current built-in behavior.

Relative implementation cost

These are relative estimates based only on public behavior; internal architecture may differ.

| Phase | Scope | Relative cost | Main risk |
|---|---|---:|---|
| 0 | Manifest schema, parsing, validation, feature flag | Small | Backward-compatible error handling |
| 1 | Bind gestures to existing first-party Codex actions | Small?Medium | Gesture/drag precedence |
| 2 | Resolve installed-app action IDs through the existing permission broker | Medium | Cross-surface authorization and unavailable actions |
| 3 | Context-menu UI, per-binding settings, managed-workspace policy | Medium | UX and admin controls |
| 4 | Optional read-only/app-published pet state and feedback animation | Medium?Large | State priority and lifecycle semantics |

A useful MVP is Phases 0?2 with only click, doubleClick, and argument-free advertised actions.

Acceptance criteria for an MVP

• Existing pets behave exactly as before.
• Invalid bindings produce a per-pet validation message rather than being silently ignored.
• An authorized, advertised test action can be bound to click or double-click.
• A denied or unavailable action never executes and falls back safely.
• Tokens, secrets, prompts, messages, commands, and file contents are not exposed to the pet manifest.
• Dragging and existing overlay interactions remain functional.
• Windows and macOS use the same manifest contract.
• The feature can be disabled by policy.

Additional information

Related issues:

• Parent interaction/API request: #21657
• Animation manifest and proof of concept: #20863
• Stay/drag interaction modes: #32377
• Click-to-toggle behavior: #32776

The goal is not to turn pet packages into executable plugins. The goal is to let the native pet act as a small, permissioned front end for actions that Codex already knows how to authorize.