Ethan Fung

The problem

The first concept prototype took blurry photos, guided visitors poorly and failed about 60% of scans. A pharmacy kiosk has no engineer standing beside it, so it had to position people reliably, recover from faults on its own and be serviceable by store staff.

What I built

• A freestanding SolidWorks enclosure (140+ parts, 28 assemblies) with 3D-printed linear rails, lightboxes and mounts.
• A three-axis camera gantry with a time-synchronized motion planner and rail-slaved pan servos, positioned to 8 mm.
• Bare-metal ESP32 firmware: a non-blocking step generator at 20k steps/s, with runtime fault detection and automatic re-homing.
• A FastAPI backend with 47 REST and WebSocket endpoints that owns one kiosk state machine: idle, positioning, scanning, analyzing, results and error.
• A dual-screen React and TypeScript interface with live guidance overlays.

Figure 1 — The gantry, the kiosk computer, and the links between them.

How 40 s became 12 s

• Analysis runs while the cameras move. Each scan submits two independent photo sets to parallel, hedged cloud requests and keeps the better-resolved result, overlapped with the gantry's next move instead of after it.
• The cameras aim themselves. A visual-servo tilt loop runs on face detections fused from two cameras (MediaPipe, with a YOLOv8-face fallback) in under 40 ms, with hysteresis gating so it never hunts. Valid captures rose from 40% to 96%.
• Guidance never contradicts itself. Signed position errors from stacked camera pairs fuse into one instruction, with minimum hold times.
• No races. Kiosk state is async but race-free: monotonic sequencing and task cancellation mean a late reply can never overwrite a newer scan.

Figure 2 — One backend-owned state machine; the screens only render it.

Results

• Scan time down 70%, from 40 seconds to 12.
• Valid-photo capture up to 96% of scans.
• Setup down from about two hours to 30 minutes.
• 1,038 automated tests, a hardware-free simulator and 30+ diagnostic tools.
• 8,000+ scans across development and pilots. The pharmacy partner asked for kiosks at three locations; those units shipped after my internship, so there are no in-store figures here.

Figure 3 — Prototype against the final kiosk; the prototype's ≈40% is derived from its ~60% failure rate.

The question

Grid-based warehouse robots need floor infrastructure, such as QR codes, tape or reflectors, before the first robot moves. Dropping the grid makes deployment faster, but robots that share free space have to avoid each other, and they can jam.

SwarmFlow asks how far decentralised coordination can go: prediction, reserved trajectories and yielding, with no central traffic controller. It answers with measurements rather than demos. Every scenario is seeded and repeatable, every run starts from fresh containers, and every metric is computed from the recorded robot states.

Architecture

Figure 1 — The fleet layer assigns orders; each robot plans and avoids the others on its own.

• The fleet layer decides who does what. Each order goes to the robot with the lowest cost: travel distance plus a congestion penalty for robots already heading to the same spot. Pick lists are ordered nearest-next, and each dock has a queue lane where a robot claims its place only when it arrives.
• Each robot decides how to get there. Every robot stamps the others into its own costmap, where they are now and where they will be over the next 2 seconds, keeps right in every aisle and plans around everything else itself. Avoidance keeps working if the fleet layer goes down.
• The same software as a real robot. ROS 2 Jazzy and Nav2 (Smac planner, MPPI controller), a custom differential-drive chassis with a 360° lidar, and physics in Gazebo Harmonic. The whole warehouse is one layout file, and docker compose up brings all 13 services up on a laptop.

How robots share space

Figure 2 — Reserve, yield, wait clear; a shared waits-for graph breaks any jam that lasts 8 s.

Each robot publishes its planned path, timed along the way and stamped with when it was reserved. A robot yields only to an older reservation that it would meet at a similar time, and it waits somewhere clear of that robot's path: never in it, and never inside a dead-end aisle. Because priority is a total order, waiting can never form a cycle.

Bodies can still jam, so there is a safety net. Every robot builds the same "who waits for whom" graph from the shared states. A cycle of any size that lasts 8 seconds is broken one robot at a time, lowest priority first: it backs up along the path it just drove, and only if nobody is behind it.

Engineering highlights

• Robots were invisible to each other. Every lidar sat at the same height, so scans passed straight through the other robots. I fixed it with a custom C++ Nav2 costmap layer that draws other robots from their shared, predicted states.
• Nav2 was silently ignoring its parameters in composed mode, running its costmaps on defaults. Found by inspecting the running stack and fixed in the launch file.
• A test-harness bug skewed the benchmarks: containers survived between runs and carried state. Caught by comparing container uptimes; all 14 affected runs were redone.
• Stalls depend on CPU pressure. The same code gave 4–21 stuck episodes per run at normal load and 23–49 on an overloaded host, so every run now records its CPU load and only matched runs are compared.
• The TF relay was a bottleneck. Porting the multi-robot TF relay to C++ cut its CPU use from 67% to 35%.
• The visualisation used too much CPU. Reading plan coordinates straight from the serialized bytes made it 15× faster per plan, tested byte-for-byte against ROS's own decoder. A light mode runs 8 robots in 0.6 of a CPU core.

Results

Figure 3 — Throughput and failed orders with coordination off and on.

• 2× throughput: in a shared narrow corridor, coordinated robots delivered 7.0 orders against 3.3 for independent navigation (n = 3).
• Prediction prevents jams: with 4 robots, predicting the others 2 seconds ahead gave 13.3 deliveries per busy 10 minutes against 11.0 without, and 0 failed orders in every run. Without prediction, one run jammed four robots at once.
• Crowded fleets don't jam: with 5 robots, reserved trajectories plus the deadlock breaker delivered 17.0 orders per 10 minutes with 0 failed, against 11.0 and 7 failed with coordination off. The shipped defaults deliver 14.7 ± 1.5 with 0 failed.
• Smoother motion: realistic acceleration limits cut heading jerk by 21% and acceleration by 17%, measured from wheel odometry at 20 Hz.
• Localisation holds under bad networks: AMCL error stays at 0.12 m with injected delay and packet loss (netem), on procedurally generated, schema-validated worlds.
• Almost no contact: across 6 busy 10-minute runs with 4 robots, two bodies touched once, by 2 cm. Pallet handling delivered 8 of 8 with no sideways docking error.
• Honest negatives: every idea was tested against a decision rule fixed before the first run, at matched CPU load. A stop-in-place yield, three anti-jamming controllers, moving pick points and a first smoothing profile all failed, and each is published with the reason.

How it was built

Eight days from an empty repository to v2.1. Contracts (message types, layout schema, fleet API) were frozen before parallel work began, and changing one needs a written decision. One CI script checks the contracts, the build and 372 tests on every change, and nothing reaches main red. Most of the code was written by AI coding agents working in parallel against tests written up front; I set the direction, made the design calls and reviewed every release.

The goal

Fold laundry with two SO-101 arms using a small task-specific policy instead of a large general model. The plan is to use a vision-language-action model (MolmoAct2) as a teacher, distill it through imitation learning, then refine the policy with DAgger and reinforcement learning. Cloth is hard because it has no fixed shape: every grasp changes the state, and a small slip early ruins the fold later.

How a policy is made

Figure 1 — One pipeline, run on harder tasks each time.

• A scripted expert (inverse kinematics plus a phase machine that reads the true cloth state) produces demonstrations. 30% of episodes are knocked off course mid-fold, so the data also shows how to recover.
• A student learns from them by behaviour cloning, either a chunk-MLP or a diffusion policy that maps a short observation history to the next 16 actions.
• DAgger closes the gap: the student drives, and the expert labels the states the student actually reaches.
• A vision student that sees only cameras and joint sensors is distilled from the privileged policy, which is the version that can run on real arms.
• Every policy is scored on held-out seed sets (easy starts, hard starts, and recovery or an arm knocked mid-fold), with confidence intervals and paired comparisons.

Why PPO failed first

PPO from scratch could not learn to fold. Digging in exposed an unreachable goal, a stalling IK solver, identical reset states and a seeding bug that ran all eight parallel environments on one identical episode, which had invalidated earlier results. A damped-least-squares IK expert, behaviour-cloned, then took corner-fold success from 0% to 83%. Potential-based reward shaping across reach, grasp, carry and place now gives RL a signal that cannot be farmed by re-grasping.

A world model that helps policies learn

A residual MLP over the cloth's physical state predicts held-out fold episodes with a corner error of 10.6 mm at ten steps, against 35.5 mm for simply assuming nothing moves. An earlier latent RSSM lost to that baseline by an order of magnitude; dropping the latent bottleneck and predicting change rather than the whole state fixed it. Training an actor on imagined rollouts through the model raised a behaviour-cloned policy from 17.5% to 87.5% success in one round, and to 95% after a round on its own collected data.

Figure 2 — Prediction error by horizon, and policy success after training in imagination.

Half fold in MuJoCo

The scripted expert sets the ceiling at 100% / 97% / 96% on the easy, hard and recovery sets (n = 200 each). The best privileged policy, a DAgger-trained diffusion policy, reached 99.5% / 77% / 79.5%, lifting recovery from 38.5%. The camera-only student reached 94% / 91% / 41%: strong from shifted starts, still weak at recovering after a knock. Offline RL (IQL) was tried three times, never beat imitation and was dropped.

Quarter fold in Isaac Sim

The project then moved to LeHome's Isaac Sim 5.1 stack, where the grasp is friction-only: nothing welds the cloth to the jaw, so it can slip, drag or fling. The quarter fold chains two folds, and the second has to pick up a strip the first one laid down.

• Releasing gently mattered most. Changing how the jaws let go lifted the expert from 41% to 92% on hard starts and from 31% to 78% with an arm knocked, on the same 100 paired seeds.
• A gripper head on the diffusion policy added 16 points.
• A flatter first fold (friction plus a lay-down move) made every success a real second fold, at no measurable cost.
• Final held-out sets (n = 100 each): the expert scores 87% / 90% / 76% and the behaviour-cloned student 66% / 59% / 38%.

Figure 3 — Quarter-fold success on the final held-out sets.

Not everything worked. A first DAgger round on the quarter fold scored 26% / 28% against the student's 49% / 39%, so it was not kept, and its cause is still open.

Engineering for speed and honest numbers

• Faster loops: CUDA graphs made diffusion sampling 12.7× faster, and bf16 with batched GPU teacher labels cut distillation time by 29%.
• More data per hour: Isaac collection runs 8 vectorized environments, 2.15× faster on Windows through D3D12, into a content-hashed, resumable dataset store.
• Grasps that are physical: friction-only grasping on PhysX particle cloth in Isaac Sim 5.1, checked by a contact audit.
• Numbers that mean something: a paired-seed harness (Wilson intervals, McNemar tests, paired bootstrap) measured a simulation noise floor of about 7 points, so smaller gains are not claimed.

Where it stands

The privileged quarter-fold result is closed. Next are the camera-only student and a success detector for the quarter fold, then sim-to-real on the physical SO-101 arms; no hardware result is claimed yet.

The question

The study, supporting the Lunar Exploration Rover System tied to NASA's Artemis program, compared analytical and deep-learning inverse kinematics for high-degree-of-freedom arms. A meaningful comparison needed real hardware running both.

What I built

• The arm: SolidWorks design, CNC machining, wiring and soldering, on CAN BLDC actuators with 14-bit encoders.
• Control: Simulink bridged to the 1 Mbps CAN bus over UDP, running MPC with inverse-dynamics feedforward on fifth-order torque trajectories.
• Perception: AprilTag and OpenCV pose estimation.
• Both IK approaches running in live control on the same setup.

The answer

On the same hardware, closed-form geometric IK reached 3 mm end-effector accuracy and a neural IK model stayed under 6 mm. Both beat the Jacobian pseudo-inverse baseline.

Demonstration

One arm is driven with a PS4 controller while a second tracks its AprilTag marker to within 1 cm.

Results

I co-authored papers in the Journal of Physics: Conference Series (2024) and IJSES (2024), and presented the system at IC-MSQUARE 2024.

The constraint

A competition entry that predicts the next step of long sequences as they stream in, from 31 GB of training data. Inference has to run on a single CPU core, so a model only counts if it is fast enough to ship.

Streaming GRU ensembles

• GRUs read each sequence step by step, trained with truncated backpropagation through time.
• Causal EMA features summarize the recent past without looking ahead.
• Distillation was part of the training recipe for ensemble members.

One CPU core

To fit the whole ensemble inside the 1-core budget, I wrote the inference path myself: a C kernel using AVX2 vector instructions and float16 arithmetic.

Learning the scoring mask

Not every step counts toward the score. An XGBoost model predicts which steps do from causal features alone, at AUC .92.

Status

The code and models are private, so there is no source link.

What I built

• An autonomy stack combining odometry and IMU fusion, PID control, multithreaded state machines and mecanum drive libraries.
• AprilTag relocalization on top of odometry and IMU fusion, cutting drift by 43%.
• Real-time OpenCV game-object detection that selects the autonomous routine at match start.
• About 50,000 lines of Java across five public seasons, with reusable drive, odometry and PID libraries.
• Swerve-drive modules in SolidWorks, and a scouting app that logged every team and match in our division for alliance selection.

Results

• World Championship #6 of 192 (2023) and #8 of 231 (2024), after 20th of 160 in 2022.
• Three Ontario championships, two Control Awards (FIRST's judged software award) and a 2nd-place Design Award.

These are team results built on everyone's work; the autonomy software was mine.

The idea

Language models are good at judging fit and bad at telling the truth about your history. In Job Appy the model may rank bullets from an evidence bank I wrote, but it never rewords them, so every line on a generated résumé traces back to something I actually did.

What it does

• Collects postings from pasted links or whole job-alert emails, with adapters for Greenhouse, Lever, Ashby, SmartRecruiters and LinkedIn and page extraction as a fallback. A bookmarklet reads WaterlooWorks postings inside the signed-in tab.
• Scores each job against the evidence bank and generates a tailored LaTeX résumé and cover letter.
• A one-page fitter compiles the LaTeX and trims the lowest-ranked bullets until the page fills exactly.

Figure 1 — From posting to a one-page résumé, with the safeguards around it.

Engineering

• Names, emails and phone numbers are scrubbed from every model request.
• Gemini runs with JSON-schema structured output. Quota exhaustion, transient errors and retired models are classified, so the app degrades to a deterministic path instead of breaking.
• A standard-library Python backend over a 15-table SQLite schema.
• Transport seams let tests run ingestion and model clients against fakes, and an automated check enforces the architecture's module boundaries.

Result

It is the tool I use for my own applications, backed by 633 tests.

Questions over messy exports

The pipeline turns CSV and XLSX CRM exports into validated records and answers natural-language questions over the normalized data.

My contribution

I designed the column-mapping contract, extractive parsing prompts and query-planning module. Teammates built the deterministic importer, deduplication, reporting and Streamlit app.

Confidence bands and validated plans

Mapping uses confidence bands: automatic above 0.90, flagged from 0.70 to 0.89 and rejected below 0.70, with a 0.90 floor for identity fields. The planner validates GPT-4o’s JSON plan before translating it into a Firestore query.

Figure 1 — Ingestion once per export; a validated plan for every question.

No model-written queries

The system keeps model suggestions inside JSON Schema and query-plan boundaries, so no raw executable query is model-written. It was a UW Data Science Club team project from January to April 2026.

How it works

MediaPipe extracts pose landmarks from the video. My analysis engine turns them into eight joint angles plus per-keypoint speed, acceleration and left–right symmetry, detects each repetition and time-warps a reference rep onto every user rep.

Figure 1 — From workout video to a coached replay, and how one rep is scored.

What I built

• Confidence pruning and temporal filtering that stabilize noisy landmarks.
• Rep scoring (70% joint form, 30% symmetry) that flags frames outside tolerance and names the three most deviating joints.
• Schema-validated coaching from Gemini 2.5 Flash through Instructor and Pydantic, grounded in a cue bank of safety principles.
• Timestamped coaching events burned into an OpenCV-annotated replay.

Figure 2 — The scored elbow angle through one clip, with the rep the pipeline detected.

FastAPI connects the pipeline to a web frontend and a Supabase-backed progress dashboard.

Limits

This is a weekend prototype for exploring exercise feedback, not a clinically validated assessment. Pose estimation and the chosen reference rep bound what the comparison can say.

What it does

Orbit Wars is a real-time strategy game where fleets capture orbiting planets, manage production and avoid comet and sun geometry.

My contribution

I built the solo heuristic agent, strategy modules and Flask plus Canvas replay visualizer. The agent uses no learned model.

Technical highlight

Seven weighted value functions score threat, safety and production. Iterative intercept aiming predicts moving targets, and segment checks reject unsafe fleet paths.

Figure 1 — One turn of the agent, with the composite value weights from the scoring code.

Results and status

The agent reliably beat Kaggle’s starter agent and ranked around the top 250 of 4,700-plus teams before the finals. It was a completed competition entry, not a production game system.

Lidar to velocity commands

The stack drives a simulated robot through Gazebo using lidar-derived obstacle costs, a remembered map, global planning and a pure-pursuit controller.

My contribution

For WATonomous onboarding, I implemented the lidar costmap, map-memory node, A* planner and pure-pursuit controller in C++ on the team’s starter template.

A planner that reroutes blocked goals

The planner uses an 8-connected occupancy grid, snaps blocked starts to free cells and reroutes unreachable goals. Tapered obstacle inflation keeps paths clear of walls.

Figure 1 — Four nodes from lidar to velocity commands, with the parameters from the code.

Simulation only

This is a simulated onboarding project. It demonstrates the navigation stack and its recovery rules; it is not a deployed autonomous vehicle.

A coach that picks the next shot

The project explored an AI badminton coach that could watch a player and eventually choose challenging shuttle placements for a robot.

My contribution

I built court-corner detection, homography mapping, player and shuttle tracking and temporal motion analysis in Google Colab.

From match footage to court coordinates

CLAHE, HSV line masks and geometry recover court coordinates. YOLOv8 pose keypoints feed a confidence-gated Kalman filter and EMA smoothing; TrackNetV3 handles shuttle tracking.

Figure 1 — From match footage to court-space tracks; the robot coach is proposed, not built.

Why there is no headline number

The work remained an exploratory team project and did not become a finished coaching product. No headline performance metric is claimed.

Self-play, labelled by Stockfish

The project trains a chess bot from self-play positions labelled by Stockfish, then uses the network inside a search procedure to choose moves.

My contribution

I built the sharded self-play data generator, ResNet policy-value trainer and bot search layer within a four-person hackathon team.

A 20,480-way policy inside alpha-beta search

Temperature and Dirichlet noise diversify generated games. The model predicts a 20,480-way move policy and clipped centipawn value, while negamax alpha-beta search uses a transposition table.

Figure 1 — Data, model and search, with the settings from the training and bot code.

How it did

The team’s competition result was weak, but the pipeline provided a substantial end-to-end training and inference experiment.

Hints in, words out, offline

The system ranks candidate words for an indirect hint while operating offline and below a one-billion-parameter model budget.

My contribution

I implemented SBERT embedding retrieval, cosine-similarity candidate ranking and alternative retrieval heads, including UMAP and KMeans experiments.

Contrastive tuning under a 1B cap

Contrastive fine-tuning improves the embedding space without relying on a large general model, keeping inference within the task constraints.

Figure 1 — Offline tuning, then retrieval under the 1B-parameter cap.

2.8× the baseline

The retrieval pipeline reached 0.83 NDCG@10 against a 0.30 baseline in the IOAI task.

Figure 2 — NDCG@10 against the baseline.

What it does

The model segments rainfall in satellite imagery where precipitation is rare and spatially sparse, making an all-background prediction deceptively competitive.

My contribution

I built the SE-ResNet encoder, U-Net decoder, patch-stitch inference and spectral re-weighting for an IOAI 2025 task.

Technical highlight

The pipeline uses 18 GOES-16 channels and emphasizes precipitation-sensitive infrared bands to counter the class imbalance.

Results and status

The model reached 0.86 IoU against a 0.20 baseline under the task’s offline compute limits.