Shippy: Engineering Trust into AI for High-Stakes Maritime Operations

When Skylight set out to build Shippy, an AI for real-time maritime awareness, they faced a stark reality: in high-stakes operations, an unreliable AI is worse than no AI at all. A wrong answer could send patrol vessels miles off course, wasting scarce resources and potentially endangering lives. The challenge wasn't the model, it was building a system they could trust to be correct, stay within its limits, and perform consistently across diverse tasks.

Skylight's approach reframes how we should think about operational AI agents:

1. Deterministic tools tame nondeterministic agents

Early Shippy prototypes constructed API calls from scratch, leading to a minefield of subtle bugs: malformed pagination silently dropping results, geometry encoding errors, and misunderstood filter types returning wrong data.

The solution? A purpose-built CLI that collapses API complexity into a predictable interface:

skylight events search --region="Panama EEZ" --activity-type=fishing --start-date=2023-05-01

This single command handles authentication, pagination, and structured output. It's self-documenting, with extensive help text and error messages that give both the agent and human developers context to recover from mistakes.

Underneath, a standardized API provides consistent operations (search and aggregate) across multiple resource types. This layering, typed API, deterministic CLI, agent skills referencing CLI commands, means each component can be tested independently, narrowing what the next layer can get wrong.

2. Isolation is non-negotiable

Skylight serves hundreds of agencies across 70+ countries. A Philippine fisheries officer's data must never leak to other users.

The solution: Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for each user session. When a conversation starts, it spins up pods packaging the agent runtime, skills, and Skylight CLI. The user's JWT is injected at provision time, scoping API calls to their data alone.

Inside this sandbox, the agent can write and run code, install dependencies, and work through multi-step analyses, all isolated at the network level and restricted to necessary services.

3. Evaluate the agent, not just the model

Most AI benchmarks use static questions, missing how an agent behaves in real workflows: selecting tools, querying live data, and knowing when to stop. Skylight built a custom eval system scoring the whole agent, model, skills, and sandbox together, against live data.

Subject-matter experts write scenarios and rubrics, choosing which criteria matter for each task. A fishing-events query, for instance, weights data accuracy most heavily, followed by boundary resolution and timeframe.

This approach captures nuances traditional benchmarks miss: How does the agent handle edge cases? Does it appropriately cite sources and explain uncertainties?

4. Skills and soul, not just silicon

Shippy isn't just a model; it's an architecture:

• Soul: A system prompt framing Shippy's persona and behavioral boundaries.
• Skills: Markdown files with structured frontmatter, defining how to handle specific requests.
• Config: Runtime settings, model selection, and secrets injection.

This modular approach keeps each component comprehensible, versioned, and easily revised. Shippy's skills cover querying the Skylight API, resolving maritime boundaries, interpreting vessel tracks, and generating interactive map links.

Crucially, Shippy's ethical boundaries are explicit in the system prompt, not hidden in fine-tuning. It won't make legal determinations or speculate beyond available data, choices that are auditable and adjustable.

The lessons from Shippy extend far beyond maritime AI. In any high-stakes domain, from healthcare to financial systems, the principles of deterministic tooling, rigorous isolation, and domain-specific evaluation are critical. Raw model power isn't enough; trust must be engineered into the system's very architecture.