The camera sees pixels. We need electrical connections.

The actual task

This is often described as breadboard classification. That name is slightly misleading. We are not trying to predict a single label such as “correct circuit.” We are trying to reconstruct the connectivity of a physical circuit from visual observations.

The camera gives us colored pixels. The checker expects something closer to an electrical graph. Between those two is the perception problem.

Input: normalized breadboard photo (roughly 1024 × 340 pixels)
  ↓
Detect: component bodies and connection endpoints
  ↓
Localize: which known holes / electrical nodes contain the endpoints?
  ↓
Associate: which endpoints belong to the same wire or component?
  ↓
Output: component records + holes + uncertainty
  ↓
Existing circuit checker evaluates electrical topology

Why the geometry helps—and where it stops helping

The standardized WB-102 breadboard has 830 physical holes, but only about 130 distinct electrical nodes in the configuration documented by the previous team. Groups of five main holes share an internal conductor, while the center groove separates two halves of the same column. The board’s hole pitch is about 2.54 mm.

The existing normalizer transforms a photograph into a canonical overhead image approximately 1024 × 340 pixels. Adjacent columns may be separated by only about 16 pixels in that view. This means a small localization error can change the inferred circuit.

There is a useful relaxation: in many cases we care about which electrical node a lead occupies, not which of five equivalent holes in that node. But the model still needs to resolve the correct column and side of the board—and exact holes may still be needed to draw a repair instruction.

One-hole error ≠ cosmetic error. If a wire is assigned to column 33 instead of column 32, the software may reconstruct a different electrical circuit and recommend a repair that isn’t needed.

Three distinct perception problems

1. Occupancy: what is plugged in?

Given the known grid, identify whether a component lead occupies each candidate location. The task becomes much easier after normalization because the expected hole centers are already known.

Challenges include low-resolution cameras, uneven lighting, angled or partly obscured connections, and dark holes that can resemble dark component legs.

2. Classification and keypoints: what part is it, and where are its leads?

A resistor is not useful to the checker until its two terminals can be located. A bounding box around the resistor body is insufficient. The previous team experimented with pose-estimation models that predict connection-point keypoints along with object detections.

The current inherited RF-DETR experiment is explicitly resistor-only: one class and two keypoints per instance. Keypoint 0 is the leftmost connection in the source image. It does not yet recognize all LEDs, wires, buttons, or sensors.

3. Association: which endpoints are connected to each other?

Suppose the detector finds six wire endpoints. It still needs to determine which endpoint pairs form the three physical wires. Wires can cross, bend, obscure one another, share colors, or disappear behind other components. The association problem is often harder than recognizing a component body.

Constraints help: component footprints, wire length classes, expected circuit structure, camera motion across multiple frames, and (when available) Arduino measurements. None should become an excuse to declare a visually unsupported connection as certain.

Known failure modes

What we inherited from the ML experiments

Hartley provided seven saved checkpoints (YOLO and RF-DETR variants) and two datasets: a real-image collection with about 170 distinct breadboard configurations and approximately 10,000 synthetic Blender examples. Many “real” image files were repeated lighting variations or augmentations of the same physical board. Training and evaluation must account for that.

A previous small evaluation reported that the RF-DETR resistor model correctly placed roughly 90% of resistors on a 30-image validation set, with an F1 increase after including synthetic data. That is a reported experiment, not an independently reproduced performance claim for the new team or a benchmark for full-board reconstruction.

Our first model baseline is the checkpoint rf-detr-704-square-norm-synth10kv0-rp170.pth, which Hartley identified as his strongest experiment. As of November 3, the new team had the checkpoint and datasets, but had not yet verified working inference using the original environment.

Turning keypoints into holes

The interface between the learned model and the deterministic checker is where experimentation becomes useful software engineering.

Resistor keypoint 1 at pixel (x1, y1)
Resistor keypoint 2 at pixel (x2, y2)
       ↓
Use the normalizer's pixel ↔ breadboard map
       ↓
Candidate holes: a14 and a19
       ↓
Component record with confidence / alternatives
       ↓
Validate using the existing Netlist schema

An illustrative component record:

{
  "id": "R1",
  "type": "resistor",
  "pins": {
    "1": { "hole": "a14" },
    "2": { "hole": "a19" }
  }
}

That record does not prove the whole board is understood. It also is not a complete netlist JSON file. The real contract and uncertainty fields must follow breadboard/netlist.py.

Why accuracy claims need care

Suppose a circuit has 34 connection points and a model reads each correctly 95% of the time. Under a simplified assumption that those errors are independent, the chance of correctly recovering all 34 is only:

0.95³⁴ ≈ 17.5%.

That independence assumption does not describe every real model, but it illustrates the issue: per-object accuracy does not equal whole-circuit reliability. A good evaluator must care about whether the electrical topology is right, not only whether individual detections look plausible.

Evaluation questions we need to answer

1. Detection: Are the expected component instances found? How many false positives?
2. Localization: How far are endpoint predictions from annotated endpoints (in pixels or hole pitches)?
3. Hole mapping: What proportion of endpoints land in the correct hole or electrical node?
4. Association: Are endpoints paired with the correct physical components?
5. Whole circuit: What fraction of boards produce an electrically correct netlist?
6. Uncertainty: When the system abstains, does it avoid false declarations that wiring is correct?
7. Runtime: Is the process fast enough for the eventual device and capture workflow?

The test set must contain unseen physical configurations, not merely alternate frames of training configurations.

The next experiment

The first engineering milestone is deliberately narrow:

Load the inherited resistor RF-DETR checkpoint, run it on a real normalized breadboard photograph, draw the predicted resistor endpoints, save the machine-readable results, and let another team member reproduce that output.

Then validate the endpoint-to-hole mapping. Only after that do we have a rational basis for choosing whether to fine-tune, extend to additional components, or change the model approach.

The research problem remains open. Our progress page will report what was actually measured, including failures.

See the current development plan · Inspect the source repository