Quanta image sensors sense light at the individual photon level and for cameras, detect them in a series of frames requiring specialized reconstruction methods. However, intensity reconstruction from quanta frames be an expensive undertaking that can severely strain the compute, memory, and latency of an edge-computing platform.
UW-Madison researchers have developed quanta neural networks (QNNs) that directly transform photon detections into downstream task objectives, without intermediate image reconstruction. A learnable temporal layer, or QNN layer, is incorporated throughout a deep neural network to yield photon-level equivalents. QNN layers are inserted between the layers of an image-based (or video-based) neural network. QNN layers use scalar state-space equations to temporally aggregate feature maps. These layers may operate at a frequency (e.g., f Hz) that differs from the quanta camera’s framerate and can be altered at inference time, owing to the QNN layer’s continuous-time formulation.
The first QNN layer used at the sensing stage is a simple instantiation of the more general layer, consisting of a single scalar recurrence that operates on photon detections (quanta frames). The sensing QNN layer adaptively accumulates photons based on the level of scene motion. Intermediate QNN layers employ multiple scalar equations, depending on the number of “heads” used, that operate on feature map tensors (spatial and channel dimensions). Downstream applications use the information as if it was a traditionally reconstructed image.
Computer vision software for photon-based imaging systems
Can be used in resource constrained-applications due to:
- Improved efficiency
- Reduced compute power
- Decreased latency
- Faster readout