Lorong AI · The One About AI × SWE · 14 October 2026
Agentic engineering at AI Singapore
Learning at the edge, bringing it home
Najib Ninaba Head of Platforms Engineering, AI Singapore
© 2026 Najib Ninaba. All rights reserved.
Hi everyone, I'm Najib. I lead Platforms Engineering at AI Singapore.
This is my first formal public talk on this topic. I've talked about it a lot inside AISG, but not on a stage like this.
The talk is from my own perspective.
Where I come from
Old HPC habits still apply
~2000
Linux HPC clusters in Singapore; Rocks tooling and packaging
2003
Co-founded Scalable Systems Pte Ltd , acquired by Platform Computing in 2006
2006–14
Platform Computing , then Revolution Analytics
2018
One of AI Singapore's original four engineers
2020
Head of Platforms Engineering
Performance and efficiency , while keeping results accurate
Scale and automate so infrastructure and tools do the heavy lifting
Split work into independent pieces that run in parallel, with map-reduce as a default pattern
Building AI platforms and tools with AI agents. Old HPC habits still apply.
I started around 2000 building Linux HPC clusters in Singapore. Then I co-founded Scalable Systems Pte Ltd, an HPC startup, which Platform Computing acquired. After that came Platform Computing and Revolution Analytics. I joined AI Singapore in 2018 as one of the first four engineers, and I've led Platforms since 2020.
When I say old HPC habits still apply, I mean this: always look for ways to improve performance and efficiency while keeping results accurate. Scale and automate so infrastructure and tools do the heavy lifting. And split work into independent pieces that run in parallel, with map-reduce as a default pattern.
I apply the same thinking to coding agents. Independent tasks fan out to agents in parallel, and the results are gathered and checked. You'll see that again later in the talk.
If asked: "This feels familiar from the HPC days. There was always a lot of work around the compute."
Platforms Engineering, AI Singapore
A small team supporting many builders
9
people in Platforms Engineering
Platforms · Infrastructure · Operations
with mentoring and education running across all of it
AIAP apprentices AI Apprenticeship Programme. Engineers trained through real project work.
AIEH engineers Delivering 100E and SIP projects with industry sponsors.
And more LADP batches, AIAP Core, AIxTech, and our PM and Admin teams.
Platforms Engineering is a nine-person team. We support seven or more teams across the Innovation Pillar.
Our main users are apprentices in our AI Apprenticeship Programme, and the engineers delivering projects with industry sponsors. We also support other programmes and our PM and Admin teams.
We run the platforms, infrastructure and operations. Mentoring and education run across all of it, and they're among the most important things we do.
Today
Four parts
01
How my practice changed From watching, to evaluating, to running several agents in parallel
02
Patterns I learned Building Orchard and contributing to RepoPrompt CE
03
Bringing it back to AISG Access, environments, plugins, mentoring and assessment
04
Where AISG is now Uneven adoption, and what we're still working out
Four parts today. First, how my own practice changed. Second, the patterns I learned while building Orchard and contributing to RepoPrompt CE. Third, how we're bringing those patterns back into AISG. And last, where AISG is now.
01
How my practice changed
2023 to today: from watching, to learning to evaluate, to running agents in parallel.
2023–2024 · Watching from a platform view
Before coding agents
If Prompt Engineering is the new hype, how long before PromptOps becomes a thing...
Mar 2023
Azure OpenAI experiments
Sep 2023
Semantic Kernel and Prompt Flow sandboxes
2024
My outlook: small fine-tuned models, open-source LLMs, OpenAI-compatible adapters , LLM infrastructure platforms
In 2023 and 2024 I was mostly watching from a platform view. In January 2023 I asked how long before "PromptOps" became a thing.
I experimented with Azure OpenAI and frameworks like Semantic Kernel and Prompt Flow. In my 2024 outlook I expected small fine-tuned models, open-source LLMs, adapters that emulate the OpenAI API on open models, and LLM infrastructure platforms.
I wasn't coding with agents yet.
Early to mid 2025 · First real use
Learning to evaluate, not vibe check
Learning to evaluate, early to mid 2025
Timeline of 2025: Claude Code in February after using Claude.ai for audit SOPs, the AI Evals For Engineers & PMs course by Hamel Husain and Shreya Shankar in June after mixed results, the first agent-written commit with a CLAUDE.md on 29 June, and a write-up on LinkedIn and X in August.
Feb 2025
Claude Code
After Claude.ai for writing audit SOPs.
Jun 2025
AI Evals For Engineers & PMs
Hamel Husain & Shreya Shankar · Maven
29 Jun 2025
First agent-written commit
With a CLAUDE.md in the repo.
Aug 2025
Wrote it up
On LinkedIn and X.
Before, I was doing ‘vibe checks’: code looked right, tests passed, ship it. Now I understand why that fails.
Stop obsessing over perfect prompts. Start by systematically analyzing WHERE and WHY they fail.
My first real use was in early 2025: Claude.ai for writing audit SOPs, then Claude Code from February.
The results were mixed. So in June 2025 I paid for the AI Evals for Engineers & PMs course by Hamel Husain and Shreya Shankar, on Maven. That changed how I worked. Before, I was doing vibe checks: the code looked right, the tests passed, ship it. Afterwards I understood why that fails, and started looking systematically at where and why things go wrong.
My first agent-written commit with a CLAUDE.md is dated 29 June 2025.
2025 to 2026 · Trying each wave
Trying the field to see what holds up
Tool waves, 2025 to 2026
Five periods of trying coding tools: exploring chat and early agents in early 2025, Claude Code arriving from February to July 2025, a broad context-and-tools phase from August to October 2025, an own harness with Pi and project skills in February 2026, and Codex with Amp runners, Devin and RepoPrompt as the main surface by mid 2026.
Early 2025
Exploring
ChatGPT
Claude desktop
OpenHands
Aider
Feb–Jul 2025
Agents arrive
Claude Code
Conductor
BMAD
Aug–Oct 2025
Context and breadth
RepoPrompt
Codex CLI
Cursor
OpenCode
Amp
Feb 2026
Own harness
Pi
30+ project skills
overnight loops
Mid 2026
Main surface
Codex
Amp runners
Devin
RepoPrompt
I put my own time and money into each wave, so I could see what holds up before bringing it to AISG.
Through 2025 I tried most of what came out. ChatGPT, Claude, OpenHands and Aider early on. Then Claude Code, Conductor and BMAD. Then RepoPrompt in August. Then Codex CLI, Cursor, OpenCode and Amp.
In February 2026 I moved to the Pi harness and stayed with it for a while.
The point wasn't to collect tools.
November 2024 to October 2026 · From my receipts
Paying to learn early
My own money on AI tools, compute and courses.
Personal spend by category
Of about US$11.9k in personal spend, about $7.9k went on models and agents, $1.9k on infrastructure and dev tools, and $2.1k on learning.
Models and agents
~$7.9k
ChatGPT Pro since Aug 2025 · Claude Max, then API credits · Devin, xAI, Warp
Infra, dev tools
~$1.9k
RepoPrompt, Exa,
Greptile, others
Learning
~$2.1k
AI Evals course,
a Maven AI course
Claude, before and after AISG's plan:
Max $200 Max $100 Pro $20
once AISG's plan covered my work
From November 2024 to now I've spent about eleven thousand nine hundred US dollars of my own money. That's roughly sixteen thousand Singapore dollars.
Most of it went on models and agents, including ChatGPT Pro since August 2025. About nineteen hundred went on infrastructure and dev tools, including the RepoPrompt lifetime licence I bought soon after the trial. About twenty-one hundred went on learning, mainly the AI evals course.
One detail: I paid for Claude Max at two hundred a month myself while there was no organisational plan. I paid to try things before there was a case for AISG to buy them. Once there was, I moved onto AISG's plan and stepped down to Max at a hundred, then to Pro.
That was my choice, to learn early. Nobody needs to spend like this to keep up.
December 2025 to March 2026 · Orchestration
Scale, once the workflow held
GitHub contributions by year
Najib's GitHub contributions were 787 in 2018 and 136 or fewer a year from 2019 to 2024, then rose to 513 in 2025 and 6,071 in 2026 up to 11 October.
GITHUB CONTRIBUTIONS BY YEAR
787
2018
25
2019
0
2020
136
2021
7
2022
96
2023
0
2024
513
2025
6,071
2026
TO 11 OCT
2.6B
tokens in 30 days, across 574 sessions
I plan with one model, implement with another, review with a third.
Model quality is improving quickly. Results still depend on workflow quality.
The step change came between December 2025 and March 2026, with Claude Code orchestrating. I'd plan with one model, implement with another, and review with a third.
In one thirty-day window I used about 2.6 billion tokens across 574 sessions.
You can see it on my GitHub. For six years I was a manager's GitHub: close to nothing. 513 contributions in 2025, and about six thousand so far this year.
My lesson from that period: model quality keeps improving, but results still depend on the quality of the workflow.
March to June 2026 · Structure
Adding structure around the agents
SPEC.md firstWritten before any code
AGENTS.md by defaultIn every new repo
Workbench repos Plans and saved exports beside each project
OpenSpec Change packages with spec deltas
Validation gates Review and checks before pushing, added in June 2026
From March to June 2026 I added structure around the agents: a spec first, AGENTS.md in every repo, workbench repos for plans, OpenSpec change packages, and validation gates before pushing. Most of these became the patterns I'll go through next, so I'll keep this short.
August 2026 to now · Fan-out
Fan-out across runners
Fan-out across runners
Najib plans, reviews and merges from Codex on a Mac Mini M2, which orchestrates Amp and Devin, with RepoPrompt supplying context. Amp runners run on Linux in AISG's on-prem infrastructure (a Proxmox VM with four H100s and an AMD W7900 workstation). Devin workers run on Macs: a Mac Studio M3 Ultra, also on-prem at AISG, and his own MacBook M3 Max.
AISG ON-PREM
RepoPrompt
context · MCP
Me, in Codex
plan · review · merge
MAC MINI M2
Amp
runners · Linux
Devin
workers · macOS
H100 VM
4 × H100 · Proxmox
W7900
AMD GPU workstation
Mac Studio
M3 Ultra
MacBook
M3 Max · my laptop
Commits by quarter
Commits in Najib's own repos grew from 13 in Q2 2025 and 27 in Q3 2025 to 345 in Q4 2025, then 1,886 in Q1 2026, 1,254 in Q2 2026 and 1,638 in Q3 2026.
COMMITS IN MY REPOS, BY QUARTER
13
Q2 25
27
Q3 25
345
Q4 25
1,886
Q1 26
1,254
Q2 26
1,638
Q3 26
Commits with agent attribution, 2026 Q2 1% Q3 13% Q4 28%
AI agents have helped me spend less time writing code directly and more time guiding the work.
Since August, Codex is my main surface. It runs on my Mac Mini and orchestrates two agent platforms, Amp and Devin, with RepoPrompt feeding it context over MCP.
I used Devin before Amp, so the Devin workers are on Macs: the Mac Studio we call mac-03, and my MacBook. Amp runners came later, on our Linux machines: a VM with four H100s and an AMD W7900 workstation. Apart from my Mac Mini and MacBook, that's AISG's on-prem infrastructure.
Independent tasks go out to separate runners, the way jobs spread across a cluster. I'll come back to that as a pattern.
The share of my commits that record an agent went from 1 percent in Q2 this year to 28 percent so far this quarter, and that undercounts, because most agent work isn't tagged. I spend less time writing code directly and more time guiding the work.
02
Patterns I learned building with agents
Orchard is where I build. RepoPrompt CE is the tool I use and help build.
Orchard · kapitan.ai ↗ · github.com/kapitan-ai/orchard ↗
Local AI on our own hardware
It started as a request to run local AI on-prem, on eight Mac Studio M3 Ultras, 512 GB each .
7mo
since the first commit
93%
of commits mine, written with agents
Orchard Console · request detail
Orchard started as a request to run local AI on our own on-prem hardware: eight Mac Studio M3 Ultras with 512 GB each.
It's one OpenAI-compatible endpoint on Apple Silicon. Each workspace has its own tokens, and every model has to be explicitly granted. It's Elixir/OTP with MLX workers. Right now it's a source-only pilot under Apache-2.0.
It has about 1,880 commits in seven months. Most of them are mine, written with agents; Luqman joined in July, and I'll come back to that.
The project site is kapitan.ai, with a demo, the docs and install steps. The source is on GitHub under kapitan-ai/orchard.
Screenshot: a synthetic demo workspace showing the request detail view.
Pattern 1 · RepoPrompt · repoprompt.com ↗ · github.com/repoprompt/repoprompt-ce ↗
Context before action
With RepoPrompt , an open-source macOS app that builds context for coding agents.
Context before action
Four steps: the codebase feeds RepoPrompt's Context Builder that picks the files fitting a fixed token budget, assumptions and a plan are reviewed before any code is written, and only then does the agent act on that context.
01
Codebase
files, CodeMaps,
tree, Git diffs
02
REPOPROMPT
Context Builder
chooses the files that
fit a fixed token budget
03
Assumptions and plan
reviewed before any
code is written
04
Agent acts
implements against
that context
I ask the coding agent to explain its assumptions before it starts writing code.
I was the first contributor besides the author : PR #2 , on 31 May 2026, the day it went open source.
Since then: Cursor agent support (#902 , #960 ) and the Homebrew cask .
The first pattern is context before action. RepoPrompt is the tool I use for this. It's an open-source, macOS context-engineering app for coding agents. Its Context Builder chooses the files that fit a fixed token budget, so the agent starts with the right context.
A habit I use every day: I ask the agent to explain its assumptions before it writes any code.
A fun fact: when RepoPrompt CE went open source on 31 May, I was the first contributor besides the author. Since then I've added Cursor agent support and maintain the Homebrew cask.
Pattern 2
Spec and plan first
$ git show --stat bb432055
bb432055 2026-03-08
init: orchard v2 — spec + agents guide
SPEC.md | 2855 ++++++++++++++++++
AGENTS.md | +++
.gitignore | +
# no code yet
Orchard's first commit held only the spec and the agent guide Since June, changes trace to the spec through OpenSpec change packages 36 ADRs record the decisions and their reasonsThe spec is now 6,000+ lines , revised in 180+ commits
The second pattern is spec and plan first. Orchard's first commit, on 8 March, contained only SPEC.md and AGENTS.md. There was no code.
Since June, changes trace back to the spec through OpenSpec change packages, and 36 architecture decision records capture why we chose what we chose. The spec has grown to more than six thousand lines.
When agents write most of the code, the spec is where I do most of my thinking.
Pattern 3
Every codebase has collaborators, and agents are among them
Collaborators on a codebase People (engineers and reviewers) and agents (Codex, Claude, Amp) both read and write the same repo, which holds AGENTS.md, SPEC.md and ADRs, a glossary and context map, repo skills, preflight gates, and tests as contracts. WRITE
READ
WRITE
READ
COLLABORATOR
People
engineers,
reviewers
COLLABORATOR
Agents
Codex, Claude,
Amp
The repo
AGENTS.md
SPEC.md · ADRs
glossary · context map
repo skills
preflight · gates
tests as contracts
Orchard started as a solo project, built with my agents. Since July, it's two of us, each with our own agents.
One onboarding guide. Orchard's CLAUDE.md just points to AGENTS.md , so every agent reads the same thing.Conventions agents can follow. RepoPrompt CE ships repo skills and a preflight every agent runs.A shared vocabulary. Glossary, ADRs and a spec both people and agents read.Tests state the contract , so an agent's change is checked like anyone's.
This one matters to me. Every codebase now has collaborators, even a small project you think of as a solo effort, because you're working on it with agents. Orchard was like that: I started it alone, with Codex, Pi, Amp and Claude Code. In July, Luqman joined AISG as a Platforms engineer, and he works on Orchard with me. Before that he was one of our AIAP apprentices, working on another Orchard project. So my practice now includes another person and his agents, as well as mine. From now on we treat agents as first-class collaborators alongside humans, and the codebase has to work for both.
In practice: one onboarding guide in AGENTS.md that every agent reads. Conventions agents can follow, like RepoPrompt CE's repo skills and its preflight script. A shared vocabulary in a glossary, ADRs and the spec. And tests that state the contract, so an agent's change is checked the same way a person's is.
We've brought this back to AISG as an AGENTS.md template.
Pattern 4
Gates and evidence
Gates and evidence
Every change to Orchard must pass five gates in order: format, compile with warnings as errors, Credo with checks for AI-style code, Dialyzer type checks, and tests with a regression test for every bug fix.
REQUIRED FOR EVERY CHANGE
01
Format
02
Compile
warnings as errors
03
Credo
plus checks for AI-style
code, from week one
04
Dialyzer
types
05
Test
a regression test
for every bug fix
181k
lines of Elixir test code
127k
lines of Elixir application code (lib/)
The 2025 evals lesson, applied to code.
Pattern four is gates and evidence. Every change to Orchard goes through format, compile with warnings as errors, Credo, Dialyzer and tests. From the first week I added checks that catch the patterns AI-written code tends to have, and every bug fix must come with a regression test.
The result is that, on the Elixir side, Orchard has more lines of test code than application code.
The same goes for models. A model qualification in Orchard stayed on hold for review, because one passing task isn't enough evidence.
For me this is the evals lesson from 2025, applied to code. Evidence first, not vibes.
Pattern 5
Second opinions from other models
Second opinions from other models
The same frozen input (context, plan or diff) goes to five models in separate lanes, Model A as the primary, and each returns its own answer; the five answers are not merged but read side by side by the engineer, who decides.
ANSWER A
ANSWER B
ANSWER C
ANSWER D
ANSWER E
Same frozen input
context, plan or diff
Model A
primary lane
Model B
separate lane
Model C
separate lane
Model D
separate lane
Model E
separate lane
I read and decide
answers side by side, no merge
December 2025: a review on a second model found critical issues in code the first model wrote. Answers stay side by side and aren't merged. X · 4 December 2025 ↗
Pattern five: get second opinions from other models. RepoPrompt's Oracle sends the same frozen input to up to five models, and their answers stay side by side. Nothing is voted on or merged. I read them and decide.
A concrete example: in December 2025, a review on a second model found a couple of critical issues in code written by the first. That's when this became a habit.
Pattern 6
Autonomy is earned, and a human approves the merge
Human in the loop first, then human on the loop. The autonomy is earned.
The agent can draft anything. I approve merges.
Orchard PR readiness packet
Exact head and base SHAs Test results and CI status Independent review, with counterexamples “An accountable human contributor has reviewed and stands behind any AI-assisted work.”
Unattended loops
Geoffrey Huntley. One agent, the same prompt, fresh context each pass.
Plan, delegate to sub-agents, verify, steer.
Iterates until the goal is judged done or the budget runs out.
Pattern six: autonomy is earned. I start with a human in the loop: I run it, watch every step and interrupt early. Once the instructions are solid, I move to a human on the loop.
This is where unattended loops come in. Geoffrey Huntley's Ralph loop, from July 2025, re-feeds one agent the same prompt with fresh context each pass, with progress kept in files and git. RepoPrompt's Orchestrate, from April 2026, is a different shape: plan, delegate to sub-agents, verify, steer. I used it extensively. Codex /goal came about three weeks later, OpenAI's version of the Ralph idea. I run these only once the instructions and gates hold, and the merge still comes back to me.
The agent can draft anything, but I approve merges. Every Orchard PR carries a readiness packet: exact SHAs, tests and CI, an independent review with counterexamples, and a checkbox confirming that a human stands behind any AI-assisted work.
Pattern 7
Scale out like a cluster
HPC cluster Job schedulerAgents · RepoPrompt CE conductor queues builds and tests, with one heavy slot per machineHPC cluster Node isolationAgents A cheap worktree for each agent, so parallel work stays isolatedHPC cluster AccountingAgents A journal of protected changes agents make through MCP
The same problems we solved on HPC clusters (scheduling, isolation and accounting), now for agents.
Pattern seven is where the HPC background shows most. Running many agents at once brings back the same problems we solved on clusters.
Scheduling: RepoPrompt CE's conductor queues builds and tests, with one heavy slot per machine, much like a job scheduler. Isolation: each agent gets its own cheap worktree. Accounting: a journal records the protected changes agents make through its MCP server.
Pattern 8
Plan for whoever runs it
AI made software cheap to build, not cheap to own.
Off by default A new experimental backend shipped behind a switch, off by default.
Fail closed Worker errors cross the boundary only as stable codes, never raw text.
Pattern six was about who approves a change. This one is about what it costs to run the system after that.
Pattern eight: AI made software cheap to build, but not cheap to own. Every system we build with agents still has to be run, secured and maintained, so we plan for whoever runs it from the start.
Two Orchard examples. A new experimental backend shipped off by default, so the people running it choose when to turn it on. And worker errors fail closed: only stable error codes cross the boundary, never raw error text.
Pattern 9
Stay model- and harness-agnostic
Bet on models and methodology, not wrappers.
MODELS
GPT Claude Opus / Sonnet GLM DeepSeek Qwen (local)
APIS
OpenAI-compatible (Orchard)
HARNESSES
Codex Claude Code Amp Devin Pi RepoPrompt (context)
My main surface so far: Claude Code, then Pi, now Codex.
Last pattern: stay model- and harness-agnostic. I use several models, through Codex, Claude Code, Amp, Devin and Pi, with RepoPrompt for context.
That's the reality: today's leading tool can fall behind, and one that's behind can come back. My own main surface went from Claude Code to Pi to Codex within a year, and Codex was one of several tools I'd tried in 2025 before it became the main one.
"Not wrappers" means harnesses: use them, but don't bet on one. What carries over is the method: spec, gates and review. Orchard speaks the OpenAI-compatible API for the same reason.
03
Bringing it back to AISG
Access, environments, plugins, mentoring and assessment. Including what didn't stick.
How it flowed back
Not everything was ready for AISG
Bringing patterns back to AISG
Timeline from July 2025 to September 2026 of patterns brought back to AISG: a Claude Code usage leaderboard and a BMAD AI/ML pack that did not stick, then the MLOps Week curriculum, a hosted Anthropic Claude Code session, a leaders workshop on AI and data infrastructure, an AGENTS.md template with the Orchard onboarding site, the Kopitiam internal Claude plugin marketplace, and LiteLLM gateways for apprentices.
2025
2026
Jul 2025
Didn't stick
cc_leaderboard
Claude Code usage, cost
Too early to roll out
Aug 2025
Didn't stick
BMAD AI/ML pack
AISG; some adoption.
Not even for me.
Aug 2025
MLOps Week
Curriculum
Oct 2025
Claude Code session
By Anthropic, hosted
at AISG
Nov 2025
Leaders workshop
AI & Data
Infrastructure
Jun 2026
AGENTS.md template
and the Orchard
onboarding site
Jul 2026
Kopitiam
Internal Claude
plugin marketplace
Sep 2026
LiteLLM gateways
For apprentices
Here's how it flowed back, including what didn't work.
In July 2025 I built a leaderboard of Claude Code usage and cost. It was too early for mass adoption, so we didn't roll it out. In August 2025 we had a BMAD agent pack for AISG. There was interest in it and some people used it, but it didn't stick, not even for me.
Other things did land: the MLOps Week curriculum, Anthropic's Claude Code session at AISG in October 2025, a leaders' workshop in November, the AGENTS.md template, Kopitiam in July this year, and LiteLLM gateways for apprentices in September.
Access to coding agents and models
Each cohort, engineer or project gets its own model access
1
Foundry account per cohort or team, programme batch, engineer sandbox or engagement
Own
keys, capacity and cost tracking for each account
Running on AISG's Azure since June 2026.
GPT-5.x GPT-6 DeepSeek Claude Haiku GLM embeddings
KSS (Kapitan Steady SelfServe) maps project roles to Azure roles
Project role Azure role Mentor Owner Apprentice Contributor Project manager Reader
For coding agents, our users typically use Claude Code through AISG's organisational plan.
For models, each group that needs its own keys, capacity and cost tracking gets its own account: an apprentice cohort or team, a programme batch, an individual engineer, or a project engagement. We've been setting these up on AISG's Azure since June.
If asked what Foundry is: Microsoft Foundry, formerly Azure AI Foundry, is Microsoft's Azure service for deploying and managing AI models. It hosts OpenAI models alongside partner and open models like DeepSeek, Claude and GLM. Each account holds the deployments for one cohort, engineer or project, with its own capacity, keys and cost tracking.
KSS, short for Kapitan Steady SelfServe, is our project and data tool. It creates a resource group per project and maps project roles to Azure roles: mentors get Owner, apprentices Contributor, project managers Reader. Access follows the project's own roles.
Project environments
How a project gets started
How a project gets started
Four separate tools set up a new project, Concierge for accounts and onboarding, KSS (Kapitan Steady SelfServe) for project records and cloud resources, git-qube-sync for repositories, and Otherside for the Coder workspace, with an operator handing off between each one because the tools do not call each other.
01
Concierge
accounts and
onboarding through
Google Chat
02
KSS
Kapitan Steady SelfServe
project records, sponsor
data, cloud resources, roles
03
git-qube-sync
repositories and
code-quality setup
04
Otherside
standard Coder workspace
and project services
Operator handoff, done by hand. The tools don’t call each other.
Sponsor's infrastructure first; our GCP if they can't provide it. Otherside provides the tooling either way.
A project starts across four separate tools, and today our operators hand off between them. Concierge handles accounts. KSS handles project records, sponsor data, cloud resources and roles. git-qube-sync sets up repositories and code-quality tooling. Otherside provides the standard Coder workspace and project services.
For 100E and SIP projects, we now ask the sponsor to provide the infrastructure. If they can't, we provide it on our GCP. Either way, Otherside provides the tooling, so the engineer's environment looks the same wherever the compute is.
LiteLLM gateways for apprentices
Models for apprentices, with a view for mentors
Otherside gateway design
Otherside model gateway Each apprentice runs Claude Code through claude-ds and a LiteLLM gateway with one personal key, which reaches one Foundry model per project, while a mentor reads usage and spend from the gateway. Claude Code
Apprentice runs claude-ds
LiteLLM gateway one personal key per apprentice
Foundry model one model per project
Mentor reads usage and spend
Status
The Otherside gateway is in a small rollout . Separately, a LiteLLM gateway for AIAP23 has run on Azure since September.
Beyond training
We also use LiteLLM in our sponsor project work.
For apprentices, we've started putting LiteLLM gateways in front of the models. This diagram is the Otherside gateway design: each apprentice gets a personal key, a command called claude-ds runs Claude Code through the gateway, and mentors can see token use and estimated spend.
That Otherside gateway is in a small rollout. Separately, we've had a LiteLLM gateway for the AIAP23 cohort running on Azure since September.
We also use LiteLLM in our sponsor project work. I can't go into those projects, because they're under the sponsors' NDAs.
Kopitiam
Our internal Claude plugin marketplace
Our internal Claude plugin marketplace
A plugin reaches Kopitiam in five steps: an AISG engineer opens a merge request, CI checks the manifest against the schema, a person on Platforms reviews and merges it, CI regenerates the marketplace catalogue, and engineers install it with the plugin install command.
01
Contributor
any AISG engineer
opens a merge request
02
CI validates
manifest checked
against the schema
03
Platforms
a person reviews
and merges
04
CI publishes
marketplace
catalogue
regenerated
05
Engineers
/plugin install
<name>@kopitiam
# in Claude Code
/plugin marketplace add git@…/aisg/kopitiam/kopitiam.git
/plugin install howmuchisit @kopitiam
Kopitiam is our internal Claude Code plugin marketplace, which my team runs. Any AISG engineer can contribute a plugin through a merge request. CI checks the manifest, someone in Platforms reviews and merges it, and CI publishes it to the marketplace.
Then anyone at AISG can add the marketplace and install the plugin in Claude Code.
What engineers have packaged
Organisational know-how, as plugins
event-speaker-panel Triage speaking invitations
howmuchisit Infrastructure cost estimates
pay-me-now Expense claims from receipts
sponsor-dd Sponsor due diligence and meeting prep
scopeskills Scoping AI and ML projects
How HowMuchIsIt is built
Python does the arithmetic. A script checks the output document.
A reviewer sub-agent sees only the report and the form; at most two fix cycles.
The author checked it against three human estimates. A fix for the miss landed the same day.
What surprised me is what engineers chose to package. Most of these aren't coding tools. They automate the organisation's operational work: triaging speaking invitations, estimating infrastructure costs, filing expense claims, sponsor due diligence, and scoping projects.
And they're built carefully. HowMuchIsIt uses Python for the arithmetic, a script to check the output, and a reviewer sub-agent that sees only the report and the form. The author checked it against three human estimates and committed a fix for the miss the same day.
That's the same evals habit, applied by one of our contributors.
Mentoring with measurement
Gauntlet: an LLM serving exercise
Each apprentice serves a model on one L4 GPU with vLLM Locust measures time to first token , throughput, latency and errors Results go to a leaderboard Checks against gaming: optional prompt fuzzing to defeat response caches, reported model IDs checked, and entries at 25% errors or more unranked
Gauntlet mentoring loop
The Gauntlet mentoring loop runs Build, then Measure (TTFT, throughput, errors), then Discuss (mentor questions and a viva), then Revise, and back to Build.
mentoring loop
Build
Measure
TTFT, throughput, errors
Discuss
mentor questions, viva
Revise
The numbers give the mentor and the apprentice something concrete to talk about.
Mentoring is core to what my team does. Gauntlet is one example: an LLM serving exercise. Each apprentice gets one L4 GPU on GCP and serves a model with vLLM. Locust measures time to first token, throughput, latency and errors, and the results go to a leaderboard.
Apprentices will optimise for whatever we measure, so we built some checks into the harness. Prompt fuzzing can make each prompt unique so a response cache doesn't help. The model IDs a server reports are checked against the expected model. And any entry with 25 percent errors or more is unranked.
If asked: fuzzing was on for AIAP-22 and off for AIAP-23; the model check compares reported IDs, not loaded weights, and a mismatch is flagged rather than unranked.
Then comes the important part: mentors discuss the results with the apprentice, ask questions, and the apprentice revises.
Agents helping us assess MLOps work
Map-reduce, applied to assessment
Map-reduce, applied to assessment
Scripts gather evidence from code, MLflow, GitLab, GCP and containers without exposing credentials to agents, about twenty Claude Code sub-agents each assess one apprentice, and mentors make the final call, deciding top scores and adjusting any score.
Scripts gather evidence
code, MLflow, GitLab,
GCP, containers
agents never see credentials
~20 sub-agents · one per apprentice
Claude Code
apprentice 1
Claude Code
apprentice 2
Claude Code
apprentice 3
Claude Code
apprentice 4
Claude Code
apprentice 5
…
Mentors decide
top scores are a mentor's call
mentors adjust any score
Agents gather evidence and do the first reading. Mentors bring the results together and make the call.
This one is map-reduce applied to assessment. For the MLOps track, scripts first gather each apprentice's evidence (code, MLflow, GitLab, GCP and containers), so the agents never see credentials.
Then Claude Code runs about twenty sub-agents in parallel, one per apprentice, over five passes. Mentors bring the results together and decide. Top scores are a mentor's call. In one run a later pass corrected an earlier finding, and a mentor adjusted a score.
It's "the agent drafts, I approve", applied to assessment.
Getting our platforms ready for agents
Agent-first where we can, with guardrails
Foundation What we have so far Still in progress MCP endpoints and agent skills Kopitiam plugins and skills; Orchard's OpenAI-compatible API MCP endpoints on KSS, Otherside and Concierge Machine-readable docs AGENTS.md template; Orchard onboarding site; AGENTS.md in our repos Platform docs agents can use directly Guardrails for people and agents KSS role mapping; per-apprentice gateway keys; Orchard's deny-by-default model grants Gateway budgets and rate limits Explicit ownership Platforms reviews every Kopitiam plugin Ownership after publish; keeping plugins in sync
If an agent has to fight its way through your wiki, the platform isn't ready.
In March I wrote about four foundations a platform needs for agents. Here's where we are against them.
What we have so far: Kopitiam and Orchard's API for skills and endpoints, the AGENTS.md template and the Orchard onboarding site for docs, KSS role mapping, per-apprentice keys and Orchard's deny-by-default grants for guardrails, and Platforms reviewing every plugin. The right column is what's still open. Two of those come back at the end.
This is still work in progress. We aim to be agent-first where possible, with the right guardrails.
LLM red-teaming in 100E projects
Red-teamed before handover
Led by SAGE , our AI Safety & Governance Engineering team
Red-teaming in the 100E lifecycle
A 100E project runs scope, build, red-team, evidence and handover, with SAGE attacking, grading and fixing the LLM app during development so the sponsor receives results anyone can re-run.
01
Scope
the sponsor's
use case
02
Build
the LLM app, in
the project repo
03
during development
Red-team
SAGE attacks, grades and fixes
04
Evidence
results anyone
can re-run
05
Handover
to the
sponsor
What it tests AI safety : harmful or off-policy answers, jailbreaks. Security : prompt injection, data leaks, tool misuse. Governance : mapped to the EU AI Act, NIST AI RMF, ISO 42001, OWASP LLM Top 10 and MITRE ATLAS.
How it works An attacker model, the project's app as target, and a grader model. /discover extracts the prompts; /attack generates tests, grades them, fixes failures and re-tests the fixes.
What it changes Each finding is documented and walked through with the project team. Fixes go at the highest layer that can control the problem: architecture, then application controls, then prompt hardening.
One more thing we bring to sponsor projects, and this is SAGE's work, our AI Safety and Governance Engineering team. LLM apps on 100E projects are red-teamed during development, before handover.
An attacker model, the app as target, and a grader model, run with two commands from the project repo. Each finding is walked through with the project team, and the fix goes at the highest layer that can control it: architecture, then application controls, then prompts.
It's the same shape as the rest of this talk: agents do the repetitive work, and the evidence is kept so people can check it.
Sponsor projects are under NDA: keep this at the process level.
04
Where AISG is now
People across AISG are picking up these tools at different paces. The platform team's job is to support each of them.
Finally, where AISG is now.
Adoption happens at different paces
Two tracks, both doing real work
A working model of the support people need, from my 2 March post
Two tracks of builders
Two tracks: about 20% deep engineering, building and maintaining the core systems, and about 80% AI-enabled builders, solving real problems with AI tools and workflow automation without waiting for engineering.
~20%
Deep engineering
build and maintain
the core systems
~80%
AI-enabled builders
solve real problems with AI tools and workflow automation, without waiting for engineering
Agents don't replace engineers. They create more builders.
1865
Jevons' paradox ↗ .The Coal Question , Jevons observed that more efficient use of coal led to more coal being used, not less. I expect something similar as building software gets cheaper.
People across AISG are picking up these tools at different paces, and that's normal. My role has given me room to try things early, and not everyone has had that. The platform team's job is to make the path easier, whatever pace people are moving at.
The pattern I keep seeing is a two-track split. Around 20 percent of people need deep engineering skills to build and maintain the core systems. The other 80 percent need to get good enough with AI tools and workflow automation to solve real problems without waiting for engineering. It's a rough split I use, not a survey. Both groups do real work, but they need different support. AIAP, 100E and AIEH sit at different points on that spectrum.
There's an industrial-era parallel I find useful: Jevons' paradox. In 1865, Jevons noticed that making coal use more efficient led to more coal being used, not less. I expect something similar with software: as building gets cheaper, more people build more things. As I wrote in March, agents don't replace engineers, they create more builders, and that means more for the platform team to support.
And from my leaderboard and BMAD: timing matters. Measurement and frameworks help once people are ready to use them.
What we're still working out
Open problems we're working on
Scaling past one person's dev environment. When anyone and their agents can implement anything, coordinating a small team is the open question.Gateway budgets and rate limits for apprentices and projects.Agent access to our platforms: MCP endpoints and machine-readable docs.
To be candid, here's what we're still working out. The one I think about most is scaling past my own dev environment. Coding agents can already do significant work, so for a small team the open question is coordination: when anyone and their agents can implement anything, how do you divide the work so people's agents don't collide? In Orchard the two of us split by area of concern, and that holds at two. Each person who joins also brings their own token budget, so more of the work can run at the same time.
On the platform side, gateways still need budgets and rate limits. And our own platforms need MCP endpoints and machine-readable docs so agents can use them directly.
Why I stay at the edge
I invest in myself to work with the newest tools and models, learn the patterns early , and bring them back to my team and AISG.
Talent development, project delivery, and platform and infrastructure practices must be built together.
So why do I do this? I invest in myself, my time and my own subscriptions, to work with the newest tools and models. That's how I learn the patterns early and bring them back to my team and AISG.
For a platform team, the practical version is that someone has to try the tools before your users need them, so you know what holds up when they arrive.
For an organisation like ours, talent development, project delivery, and platform and infrastructure practices have to be built together. Staying at the edge is how I make sure the platform side keeps up.
What carries across
Four things I'd take anywhere
Context and a plan first before the agent acts
Write for every collaborator people and agents
Evidence before acceptance evaluate, don't vibe check
An accountable owner for everything that ships
To close, four things I'd take anywhere. Context and a plan before the agent acts. Write every codebase for all its collaborators, people and agents. Evidence and evaluation before anything is accepted. And an accountable owner for everything that ships.
Q&A
Questions and discussion
My practice · Orchard and RepoPrompt CE · how AISG supports its builders
Happy to take questions on any of this: my own practice, Orchard or RepoPrompt CE, or how we support our engineers and apprentices.
Annex slides follow the contact slide if needed.
Annex · Orchard architecture
Orchard, in one picture
Orchard, in one picture Clients with workspace tokens call the Elixir controller, which keeps its state in Postgres and drives one node agent per Mac over mTLS gRPC, and each node agent runs Python MLX workers on Apple Silicon. ADVISORY LOCKS
Clients apps and coding agents with workspace tokens
Controller Elixir/OTP, Phoenix LiveView Console /v1/responses, /v1/chat/completions
Node agent one per Mac mTLS gRPC
MLX workers Python Apple Silicon
Postgres the only store
State
Postgres is the only store, with advisory locks. No Kubernetes, Redis or broker.
Access
Deny-by-default model grants; tool calls are returned to the client, never run on the server.
Status
0.5.0-dev pilot. Single-node path today; multi-node scheduling on the roadmap.
Backup for architecture questions. Clients call the controller with workspace tokens. The controller is Elixir/OTP with a Phoenix LiveView Console. A node agent on each Mac runs MLX workers over mTLS gRPC. Postgres is the only store. Today it's a single-node pilot, with multi-node scheduling on the roadmap.
Annex · Model access on Azure, June to October 2026
Where model access sits
Training programmes
Accounts per AIAP cohort and team, and per programme batch
Engineer sandboxes
Per-engineer accounts with their own model mix
Engagements
Red-teaming and scoping environments, one account each
Platform team
Load tests, reproduction environments, Otherside devbox
GPT-5.x GPT-4.x / 4o embeddings DeepSeek GPT-6 o-series GLM Claude Haiku
Backup for questions on model access. Since June we've set up Foundry accounts across training programmes, engineer sandboxes, engagements and the platform team. Most deployments are GPT-5.x and GPT-4.x, with embeddings, DeepSeek and GPT-6, and a few on GLM and Claude Haiku.
Annex · The infrastructure we provide
Our own fleet, plus three clouds
H100
A few dozen H100 GPUs on a Proxmox cluster
8
Mac Studio M3 Ultras, 512 GB unified memory each
Azure · AWS · GCP
For 100E and SIP, the sponsor's infrastructure comes first
Earlier, at NUS
We once ran close to 300 nodes from a small OCP server farm, mainly for compute and storage. We decommissioned it when AISG moved to NTU.
Backup for questions about our hardware.
Here's what we run. On-prem, a Proxmox cluster with a few dozen H100s, eight Mac Studio M3 Ultras with 512 GB each, and hundreds of terabytes of storage. In the cloud: Azure, AWS and GCP. For 100E and SIP projects we ask the sponsor to provide the infrastructure first.
A bit of history: when we were at NUS, we once ran close to 300 nodes from a small OCP server farm, mainly for compute and storage. We decommissioned it when AISG moved to NTU. We've always run our own fleet at scale.
The same hardware runs our users' work, Orchard, and my own agent runners.
