Can one exact gradient teach a microscope to refocus itself and improve the biological analysis at the end of the pipeline?
Hii guys…
I’m Raj Patil,
For the Tesseract Hackathon 2026, Track 05: Differentiable graphics and rendering,
I built TessScope, an in-silico closed-loop fluorescence microscopy research prototype.
The project explores three connected questions:
Can a microscope recognize that it has lost focus?
Can it decide how far and in which direction to move?
Can the final biological analysis teach the optics at the beginning?
Click on the thumbnail
to watch the system demo on YouTube:
The real scientific problem :
During long or high-throughput microscopy experiments, focus can shift because of stage-positioning errors, uneven sample height, and thermal or mechanical drift.
The resulting blur is not only a visual problem. It can weaken automated cell detection and segmentation, force repeated imaging, and increase manual intervention.
This matters for automated microscopy, high-content screening, computational optical design, bioimage analysis, and quantitative biology.
This affects communities working in:
-
Automated and high-content microscopy
-
Computational microscopy and optical design
-
Bioimage analysis
-
Cell screening and quantitative biology
-
Autonomous scientific instruments
A useful autofocus system therefore needs to do more than detect blur. It must determine the direction and magnitude of a correction while preserving the biological structures needed for downstream analysis.
What TessScope does :
TessScope connects the complete decision loop:
First exposure → signed-defocus estimate → bounded stage action → corrected exposure → nucleus-analysis objective
Within the simulated imaging system:
- A learned phase-only pupil forms the first fluorescence exposure.
- Autofocus estimates the magnitude and direction of defocus.
- The controller converts that estimate into a bounded stage correction.
- The same optical system forms a second exposure at the remaining defocus.
- A frozen nucleus observer evaluates the biological usefulness of the images through a task loss.
- An exact gradient flows backward through the complete loop to update the learned pupil parameters.
The important idea is that TessScope does not optimize image sharpness in isolation. The final biological-analysis objective influences the learned optics through the autofocus decision and stage action.
Why Tesseract is essential to the architecture:
The pipeline crosses real computational boundaries:
JAX/Chromatix optics → NumPy/SciPy autofocus → JAX/Chromatix correction → PyTorch/InstanSeg observer
Three served Tesseracts participate in the loop. The optics component is called twice, before and after the predicted correction.
These components naturally belong to different scientific ecosystems. Reimplementing them in a single framework would create substantial engineering work and make it harder to retain the specialized tools researchers already use.
Passing images between libraries is not enough: the reverse derivative must also survive those boundaries.
Tesseract allows each component to remain in its native stack while exposing the complete composition as one differentiable program.
The reverse path combines native JAX differentiation, custom analytic and implicit derivatives through the SciPy autofocus calculation, and PyTorch input gradients from the frozen observer.
This connects optical design, autofocus decision-making, stage correction, and biological analysis without rewriting everything in one framework.
What the experiments showed:
Using real BBBC006 fluorescence microscopy source images within a simulated image-formation and correction loop, the primary hard-density validation subset produced:
| Metric | Reported result |
|---|---|
| Panoptic quality improvement after one closed-loop correction | +0.0601 PQ |
| Signed focus-direction accuracy | 98.89% |
| Mean remaining absolute focus error | 1.27 µm |
| Corrected PQ improvement over the matched stopped-gradient control | +0.006259 |
| 95% paired-well bootstrap interval for that comparison | [+0.000670, +0.011547] |
PQ means panoptic quality, measured on a 0 to 1 scale. It is not percent accuracy.
Does learning through the autofocus action actually help?
At matched parameter values, the exact-gradient and stopped-gradient versions have identical forward computations.
The difference is in optimization: the control blocks the gradient through the stage action, while the exact-gradient version allows it to pass.
The reported performance comparison is between the resulting optimized designs.
The positive paired-well bootstrap interval provides evidence that learning through the autofocus action contributes useful improvement in this experiment, rather than merely demonstrating that an autofocus component exists.
Scope and limitations:
TessScope is currently an in-silico research prototype. These are validation-only results, not tests on a physical microscope. The held-out BBBC006 test split remains sealed.
The reference instances are automated CellProfiler-derived annotations, not manual biological ground truth.
I do not claim that exact optimization beats every alternative. Although it improved over the matched stopped-gradient control. The repository documents these limitations, additional evaluation constraints, and negative follow-up experiments.
How the scientific community can build on it
TessScope is designed as a reusable research foundation rather than a closed demonstration.
Researchers can replace or extend:
- The optics: the simulator, pupil representation, aberration model, or sensor calibration.
- The autofocus and control: the estimator, control policy, actuator constraints, or acquisition sequence.
- The biological analysis: the frozen observer, segmentation task, or scientific objective.
- The experimental setup: the imaging modality or, after appropriate calibration and safety validation, a physical microscope-stage interface.
The longer-term goal is task-aware scientific imaging: systems whose final analysis helps improve how they acquire data.
Potential applications include high-content screening, long-duration cellular experiments, adaptive imaging, and other autonomous scientific instruments.
Physical microscope integration remains future work, not a capability demonstrated by the current results.
Code:github
The repository includes the implementation, experimental protocol, derivative checks, frozen results, and full reproduction instructions.
A lightweight cached-evidence replay is also available for quick inspection. It is explicitly a replay of recorded evidence, not live inference or retraining.
A special thanks to Alexander Lavin, M. (mapo), Dion Häfner, Jonathan Brodrick, and the Pasteur Labs & ISI team for organizing the hackathon and supporting the office hours.
I’d especially welcome feedback on the cross-framework gradient design and the experiments needed before moving this loop onto a physical microscope.
Built by Raj Patil for the Tesseract Hackathon 2026.