Deep Learning Models¶

evlib.models ships two PyTorch model families for event camera data: E2VID (event-to-video reconstruction) and RVT (event-based object detection). Both are implemented as plain PyTorch modules with pretrained-weight loading and GPU inference, and both share a common base interface for turning raw events into model input.

Installing PyTorch support¶

evlib.models requires PyTorch, which is not part of the default install:

pip install evlib[torch]
# or, an identical alias:
pip install evlib[pytorch]

Both extras resolve to the same pin (torch >= 2.0.0, torchvision >= 0.15.0). If PyTorch is not installed, importing a name such as evlib.models.E2VID raises an ImportError that names the missing dependency and the install command, rather than a bare AttributeError.

The common `BaseModel` interface¶

E2VID, E2VIDRecurrent and RVT all subclass evlib.models.base.BaseModel. The base class fixes three things every model shares: how a model is constructed, how raw events are turned into a dense input tensor, and the shape of the abstract reconstruct method each subclass must implement.

```python notest class BaseModel(ABC): def init(self, config: Optional[ModelConfig] = None, pretrained: bool = False): ...

def preprocess_events(self, events, height=None, width=None) -> tuple:
    """Accepts a structured array, an (xs, ys, ts, ps) tuple, or a Polars
    LazyFrame/DataFrame; returns (xs, ys, ts, ps, height, width)."""

def events_to_voxel_grid(self, xs, ys, ts, ps, height, width) -> np.ndarray:
    """Builds a (num_bins, height, width) voxel grid via
    evlib.representations.create_voxel_grid."""

@abstractmethod
def reconstruct(self, events, height=None, width=None) -> np.ndarray: ...

``preprocess_eventsis the common entry point: pass it a structured numpy array withx/y/t/pfields, a(xs, ys, ts, ps)tuple, or a PolarsLazyFrame/DataFrame(the kindevlib.load_eventsreturns), and it normalises all three shapes to(xs, ys, ts, ps, height, width)numpy arrays, inferringheight/widthfrom the max coordinates when not given.events_to_voxel_gridbuilds on that to produce a dense(num_bins, height, width)tensor using evlib's own voxel-grid representation;E2VIDoverrides it with a simpler binning scheme better suited to its architecture (see below). Every model also implements_build_model()and_load_pretrained_weights()(both abstract onBaseModel), and exposesconfig(aModelConfig) andpretrained` (bool) as public attributes.

`ModelConfig`¶

evlib.models.config.ModelConfig is a dataclass with the fields in_channels (default 5), out_channels (1), base_channels (32), num_layers (4), num_bins (5), use_gpu (True), and a free-form extra_params dict for architecture-specific extensions. A small set of named presets ship in CONFIGS and are reachable through get_config: python from evlib.models.config import get_config cfg = get_config("lite") # matches the bundled e2vid-lite.pth architecture print(cfg)

The preset names are default, lite, high_res, fast, temporal, spade, and ssl; get_config raises ValueError for anything else.

E2VID: single-frame reconstruction¶

E2VID is a UNet (encoder/residual-bottleneck/decoder with skip connections, skip_type="sum" or "concat") based on the official RPG E2VID model (Rebecq et al., CVPR 2019). reconstruct(events, height=None, width=None) preprocesses the events, bins them into a voxel grid, and runs one forward pass, returning a single (height, width) float32 frame:

import evlib
from evlib.models import E2VID

events = evlib.load_events("data/slider_depth/events.txt")

# pretrained=False builds the architecture with random weights; useful for
# checking the API and tensor shapes without a checkpoint on disk.
model = E2VID(pretrained=False)
frame = model.reconstruct(events)
print(frame.shape, frame.dtype)  # (180, 240) float32

E2VID's constructor also takes skip_type, num_encoders (default 4), num_residual_blocks (default 2), and norm ("BN", "IN", or None), matching the original RPG defaults. E2VID overrides events_to_voxel_grid with a simple temporal-binning accumulation (no bilinear interpolation), which is what the original E2VID architecture expects as input, rather than evlib's more general voxel-grid representation.

`E2VIDRecurrent`: stateful reconstruction¶

E2VIDRecurrent is the recurrent variant matching the bundled e2vid-lite.pth checkpoint: every encoder is followed by a ConvLSTM, so reconstruct takes and returns a state list, letting you reconstruct a video sequence chunk by chunk:

```python notest import evlib from evlib.models import E2VIDRecurrent

events = evlib.load_events("data/slider_depth/events.txt")

model = E2VIDRecurrent(pretrained=False) frame, state = model.reconstruct(events) print(frame.shape) # (180, 240)

Feed `state` back in on the next chunk of the same sequence:¶

next_frame, state = model.reconstruct(next_events_chunk, state=state)

Unlike `E2VID`, its `pretrained=True` path always loads the exact bundled `e2vid-lite.pth` checkpoint and derives the architecture (`num_bins`, `base_num_channels`, `num_encoders`, `num_residual_blocks`, `norm`) from the checkpoint's own tensor shapes before loading it `strict=True`.

## RVT: event-based object detection

`RVT` combines a recurrent MaxViT-style backbone (`RVTBackbone`), a PAFPN neck (`PAFPN`), and a YOLOX detection head (`YOLOXHead`) into a single detector, following "Recurrent Vision Transformers for Object Detection with Event Cameras" (Gehrig & Scaramuzza, CVPR 2023). Configuration lives in `RVTModelConfig`, which extends `ModelConfig` with detection-specific fields: `model_variant` (`"tiny"`/`"small"`/`"base"`), `temporal_bins` (default 10), `num_classes`, `confidence_threshold`, `nms_threshold`, `max_detections`, `input_height`/`input_width`, and `fpn_depth_multiplier`. `RVTModelConfig.tiny()`, `.small()`, and `.base()` are ready-made presets that set `base_channels` and `fpn_depth_multiplier` for each variant.

```python
from evlib.models import RVT

model = RVT(variant="tiny", pretrained=False, num_classes=2)
print(model.variant, model.temporal_bins)  # tiny 10
print(model.config.confidence_threshold, model.config.nms_threshold)  # 0.1 0.45

The detection entry point is detect(events, height=None, width=None, confidence_threshold=None, nms_threshold=None, reset_states=False), which preprocesses events into a stacked-histogram tensor (2 * temporal_bins channels), runs the backbone/FPN/head, and returns a list of dicts each shaped {"bbox": [x1, y1, x2, y2], "score": float, "class": int, "class_name": str}. RVT also implements BaseModel.reconstruct for interface compatibility, but since RVT is a detector rather than a reconstruction model, that method just draws the detected boxes onto a blank (height, width) image; call detect directly for real use.

```python notest

The MaxViT partition size must match the padded input resolution the¶

checkpoint was trained with (see partition_size_from_hw in¶

evlib.models.rvt_backbone; the RVT pipeline guide covers gen4's (6, 10)¶

partition for 384x640 padded input in detail).¶

detections = model.detect(events, height=720, width=1280, reset_states=True) for det in detections: print(det["class_name"], det["score"], det["bbox"]) ```

Because RVT is recurrent, detect carries LSTM state across calls by default; reset_states() clears it explicitly, and set_worker_id(worker_id) lets a multi-worker evaluation loop keep one state per worker (see Evaluation, which uses exactly this to stream gen4 validation sequences).

GPU inference¶

Every model picks a device at construction time: self._device = torch.device("cuda" if torch.cuda.is_available() else "cpu"). There is no separate .cuda() call needed for the common path; RVT.to(device) overrides nn.Module.to to keep self._device in sync if you move the model explicitly.

Loading pretrained weights¶

Weight files are not distributed with the pip install evlib[torch] wheel; python/evlib/models/weights/ is gitignored, so you provide checkpoints yourself before passing pretrained=True:

E2VID looks for any *.pth* file in weights/, picks the first one alphabetically, remaps the reference RPG E2VID checkpoint keys onto its own module names, and adapts its architecture (base channels, encoder count, norm, transposed-conv vs. upsample-conv) to whatever the checkpoint's tensor shapes imply.
E2VIDRecurrent looks for exactly weights/e2vid-lite.pth and loads it strict=True.
RVT looks for weights/rvt-{t,s,b}.ckpt, weights/rvt_{tiny,small,base}.ckpt, or weights/rvt.ckpt (checked in that order for the requested variant), remaps the PyTorch Lightning checkpoint's parameter names, and loads with strict=False.

If pretrained=True and no matching file is found, or loading otherwise fails, all three raise (FileNotFoundError or RuntimeError) rather than silently continuing with the freshly-initialised random weights: a model that looks pretrained but scores a meaningless mAP is worse than one that fails loudly at construction time.

Where to go next¶

Datasets & Training Data: evlib.data's PyTorch datasets, augmentation, and the evlib-rvt-preprocess console script for turning raw sequences into RVT-compatible training data.
Evaluation: Prophesee-compatible mAP scoring for a trained RVT model.
RVT Preprocessing Pipeline: the four scatter-add backends behind the stacked-histogram representation RVT consumes.