200 lines
8.7 KiB
Markdown
200 lines
8.7 KiB
Markdown
# Buffer Sharing Design: Video → HAL
|
||
|
||
## Problem
|
||
|
||
We need to inject decoded video frames into the camera HAL's output buffer pipeline at the HAL process level (camera provider APEX), replacing real camera frames with virtual video frames.
|
||
|
||
## Architecture Context
|
||
|
||
```
|
||
[App Process] [Camera Provider APEX Process]
|
||
┌─────────────────┐ ┌──────────────────────────────────┐
|
||
│ ExoPlayer │ │ libgooglecamerahal.so │
|
||
│ ↓ decodes │ │ ↓ processCaptureRequest() │
|
||
│ MediaCodec │ │ ↓ fills output buffers │
|
||
│ ↓ YUV frames │ │ ↓ (AHardwareBuffer/gralloc) │
|
||
│ VirtualCamera │ │ cameraserver reads via binder │
|
||
│ Renderer │ └──────────────────────────────────┘
|
||
│ ↓ renders to │ ↑
|
||
│ Surface │ │ WE INJECT HERE
|
||
└─────────────────┘ ┌──────────────────────────────────┘
|
||
│ libcamera_hook.so (LD_PRELOAD) │
|
||
│ intercepts processCaptureRequest │
|
||
└──────────────────────────────────┘
|
||
```
|
||
|
||
## Evaluated Approaches
|
||
|
||
### 1. AHardwareBuffer Direct Write (CHOSEN)
|
||
|
||
**How it works**: HAL output buffers are `buffer_handle_t` backed by gralloc/AHardwareBuffer. Lock the buffer for CPU write, copy YUV frame data, unlock.
|
||
|
||
**Pros**:
|
||
- No extra memory allocation needed (reuse HAL's buffer)
|
||
- Direct write to the buffer cameraserver will read
|
||
- Works with YUV_420_888 (format 32) which is CPU-accessible
|
||
- Pixel 9a (API 36) fully supports AHardwareBuffer_lock()
|
||
|
||
**Cons**:
|
||
- CPU copy required (not zero-copy)
|
||
- IMPLEMENTATION_DEFINED (format 35) buffers may not be CPU-lockable
|
||
- Must handle buffer stride/alignment correctly
|
||
|
||
**Verdict**: **CHOSEN** for YUV_420_888 preview streams. This is the primary virtual camera use case.
|
||
|
||
### 2. Shared Memory (ashmem/memfd) Cross-Process
|
||
|
||
**How it works**: App process decodes video, writes YUV frames to shared memory. Provider process reads from shared memory and copies into HAL buffers.
|
||
|
||
**Pros**:
|
||
- Clean separation: decoder in app, injection in provider
|
||
- App has full MediaCodec/ExoPlayer access
|
||
- memfd_create() available on Android 11+ (API 30+)
|
||
|
||
**Cons**:
|
||
- Extra copy: decoder → shared memory → HAL buffer
|
||
- Cross-process synchronization complexity
|
||
- Need IPC mechanism (binder, socket, or signal)
|
||
|
||
**Verdict**: **CHOSEN** as the transport mechanism from app to provider process. Combined with Approach 1 for the final write.
|
||
|
||
### 3. ION/dmabuf
|
||
|
||
**How it works**: Allocate ION memory, pass dmabuf fd to both decoder and HAL.
|
||
|
||
**Pros**:
|
||
- Zero-copy potential
|
||
- GPU/ISP accessible
|
||
|
||
**Cons**:
|
||
- Vendor-specific ION heap configurations
|
||
- Requires /dev/ion access (SELinux restrictions in provider process)
|
||
- Complex buffer lifecycle management
|
||
- Overkill for our use case
|
||
|
||
**Verdict**: **REJECTED**. Too complex, vendor-dependent, and SELinux-hostile.
|
||
|
||
## Chosen Architecture
|
||
|
||
### Two-Stage Pipeline
|
||
|
||
```
|
||
Stage 1: App Process (Video Decoding)
|
||
┌─────────────────────────────────────┐
|
||
│ ExoPlayer → MediaCodec │
|
||
│ ↓ decoded YUV420 frames │
|
||
│ FrameRingBuffer (shared memory) │
|
||
│ ↓ memfd + mmap │
|
||
│ SharedMemoryWriter │
|
||
└─────────────────────────────────────┘
|
||
│
|
||
│ memfd (fd passed via binder/property)
|
||
↓
|
||
Stage 2: Provider Process (HAL Injection)
|
||
┌─────────────────────────────────────┐
|
||
│ SharedMemoryReader │
|
||
│ ↓ reads latest YUV frame │
|
||
│ BufferConverter │
|
||
│ ↓ handles format/stride conversion │
|
||
│ AHardwareBufferWriter │
|
||
│ ↓ lock → memcpy → unlock │
|
||
│ HAL output buffer (to cameraserver) │
|
||
└─────────────────────────────────────┘
|
||
```
|
||
|
||
### Buffer Lifecycle
|
||
|
||
1. **Allocation**: HAL allocates output buffers during `configureStreams()`. We observe and record buffer dimensions, format, and stride.
|
||
|
||
2. **Frame Production**: App process decodes video via MediaCodec, writes YUV frames to a ring buffer in shared memory (memfd). Each frame has a sequence number and timestamp.
|
||
|
||
3. **Frame Consumption**: In `processCaptureRequest()`, our hook:
|
||
a. Reads the latest frame from shared memory
|
||
b. Locks the HAL output buffer via `AHardwareBuffer_lock()` (for YUV_420_888)
|
||
c. Copies frame data with stride conversion if needed
|
||
d. Unlocks the buffer
|
||
e. Returns to cameraserver (appears as real camera frame)
|
||
|
||
4. **Synchronization**:
|
||
- Shared memory ring buffer uses atomic sequence numbers
|
||
- Reader always grabs the latest complete frame (no blocking)
|
||
- If no frame available, forward to real HAL (passthrough)
|
||
|
||
### Format Handling
|
||
|
||
| Stream Format | Strategy |
|
||
|---|---|
|
||
| YUV_420_888 (32) | Direct AHardwareBuffer_lock + memcpy |
|
||
| IMPLEMENTATION_DEFINED (35) | Passthrough to real HAL (opaque GPU format) |
|
||
| JPEG/BLOB (37) | Passthrough to real HAL (encode handled by HAL) |
|
||
|
||
### Memory Layout
|
||
|
||
Shared memory ring buffer (memfd):
|
||
```
|
||
┌─────────────────────────────────────────────┐
|
||
│ Header (256 bytes) │
|
||
│ - magic: 0xCSWAPR00 │
|
||
│ - frame_count: uint32 │
|
||
│ - write_index: atomic<uint32> │
|
||
│ - read_index: atomic<uint32> │
|
||
│ - frame_width: uint32 │
|
||
│ - frame_height: uint32 │
|
||
│ - frame_size: uint32 │
|
||
├─────────────────────────────────────────────┤
|
||
│ Frame 0 (width × height × 3/2 bytes) │
|
||
│ - Y plane: width × height │
|
||
│ - U plane: width/2 × height/2 │
|
||
│ - V plane: width/2 × height/2 │
|
||
├─────────────────────────────────────────────┤
|
||
│ Frame 1 ... │
|
||
├─────────────────────────────────────────────┤
|
||
│ Frame N-1 ... │
|
||
└─────────────────────────────────────────────┘
|
||
```
|
||
|
||
Ring size: 4 frames (enough for 30fps video with slight timing jitter).
|
||
|
||
### AHardwareBuffer Write Procedure
|
||
|
||
```cpp
|
||
// In processCaptureRequest hook:
|
||
for (int i = 0; i < request->output_buffer_count; i++) {
|
||
auto& buf = request->output_buffers[i];
|
||
auto stream = find_stream(buf.stream_id);
|
||
|
||
if (stream->format == FORMAT_YUV_420_888 && g_hook_state.virtual_camera_enabled) {
|
||
// Lock buffer for CPU write
|
||
AHardwareBuffer* ahb = AHardwareBuffer_fromNativeHandle(buf.handle);
|
||
AHardwareBuffer_Desc desc;
|
||
AHardwareBuffer_describe(ahb, &desc);
|
||
|
||
void* cpu_addr = nullptr;
|
||
int result = AHardwareBuffer_lock(ahb,
|
||
AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN,
|
||
-1, nullptr, &cpu_addr);
|
||
|
||
if (result == 0 && cpu_addr) {
|
||
// Copy YUV frame from shared memory
|
||
copy_yuv_frame_to_buffer(cpu_addr, desc.stride, desc.height,
|
||
g_frame_reader->latest_frame());
|
||
AHardwareBuffer_unlock(ahb, nullptr);
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
## Implementation Tasks
|
||
|
||
- **Task 7**: Video decoder (MediaCodec NDK) in app process
|
||
- **Task 9**: Buffer format conversion + shared memory ring buffer
|
||
- **Task 6**: processCaptureRequest integration (AHardwareBuffer write)
|
||
|
||
## Pixel 9a Specifics
|
||
|
||
- API level: 36 (Android 16)
|
||
- AHardwareBuffer: fully supported
|
||
- Gralloc: `/vendor/lib64/hw/gralloc.gs101.so` (Tensor G3)
|
||
- YUV_420_888: CPU-accessible, linear layout
|
||
- IMPLEMENTATION_DEFINED: opaque, GPU-only (passthrough)
|