Initial commit
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# Buffer Sharing Design: Video → HAL
|
||||
|
||||
## Problem
|
||||
|
||||
We need to inject decoded video frames into the camera HAL's output buffer pipeline at the HAL process level (camera provider APEX), replacing real camera frames with virtual video frames.
|
||||
|
||||
## Architecture Context
|
||||
|
||||
```
|
||||
[App Process] [Camera Provider APEX Process]
|
||||
┌─────────────────┐ ┌──────────────────────────────────┐
|
||||
│ ExoPlayer │ │ libgooglecamerahal.so │
|
||||
│ ↓ decodes │ │ ↓ processCaptureRequest() │
|
||||
│ MediaCodec │ │ ↓ fills output buffers │
|
||||
│ ↓ YUV frames │ │ ↓ (AHardwareBuffer/gralloc) │
|
||||
│ VirtualCamera │ │ cameraserver reads via binder │
|
||||
│ Renderer │ └──────────────────────────────────┘
|
||||
│ ↓ renders to │ ↑
|
||||
│ Surface │ │ WE INJECT HERE
|
||||
└─────────────────┘ ┌──────────────────────────────────┘
|
||||
│ libcamera_hook.so (LD_PRELOAD) │
|
||||
│ intercepts processCaptureRequest │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Evaluated Approaches
|
||||
|
||||
### 1. AHardwareBuffer Direct Write (CHOSEN)
|
||||
|
||||
**How it works**: HAL output buffers are `buffer_handle_t` backed by gralloc/AHardwareBuffer. Lock the buffer for CPU write, copy YUV frame data, unlock.
|
||||
|
||||
**Pros**:
|
||||
- No extra memory allocation needed (reuse HAL's buffer)
|
||||
- Direct write to the buffer cameraserver will read
|
||||
- Works with YUV_420_888 (format 32) which is CPU-accessible
|
||||
- Pixel 9a (API 36) fully supports AHardwareBuffer_lock()
|
||||
|
||||
**Cons**:
|
||||
- CPU copy required (not zero-copy)
|
||||
- IMPLEMENTATION_DEFINED (format 35) buffers may not be CPU-lockable
|
||||
- Must handle buffer stride/alignment correctly
|
||||
|
||||
**Verdict**: **CHOSEN** for YUV_420_888 preview streams. This is the primary virtual camera use case.
|
||||
|
||||
### 2. Shared Memory (ashmem/memfd) Cross-Process
|
||||
|
||||
**How it works**: App process decodes video, writes YUV frames to shared memory. Provider process reads from shared memory and copies into HAL buffers.
|
||||
|
||||
**Pros**:
|
||||
- Clean separation: decoder in app, injection in provider
|
||||
- App has full MediaCodec/ExoPlayer access
|
||||
- memfd_create() available on Android 11+ (API 30+)
|
||||
|
||||
**Cons**:
|
||||
- Extra copy: decoder → shared memory → HAL buffer
|
||||
- Cross-process synchronization complexity
|
||||
- Need IPC mechanism (binder, socket, or signal)
|
||||
|
||||
**Verdict**: **CHOSEN** as the transport mechanism from app to provider process. Combined with Approach 1 for the final write.
|
||||
|
||||
### 3. ION/dmabuf
|
||||
|
||||
**How it works**: Allocate ION memory, pass dmabuf fd to both decoder and HAL.
|
||||
|
||||
**Pros**:
|
||||
- Zero-copy potential
|
||||
- GPU/ISP accessible
|
||||
|
||||
**Cons**:
|
||||
- Vendor-specific ION heap configurations
|
||||
- Requires /dev/ion access (SELinux restrictions in provider process)
|
||||
- Complex buffer lifecycle management
|
||||
- Overkill for our use case
|
||||
|
||||
**Verdict**: **REJECTED**. Too complex, vendor-dependent, and SELinux-hostile.
|
||||
|
||||
## Chosen Architecture
|
||||
|
||||
### Two-Stage Pipeline
|
||||
|
||||
```
|
||||
Stage 1: App Process (Video Decoding)
|
||||
┌─────────────────────────────────────┐
|
||||
│ ExoPlayer → MediaCodec │
|
||||
│ ↓ decoded YUV420 frames │
|
||||
│ FrameRingBuffer (shared memory) │
|
||||
│ ↓ memfd + mmap │
|
||||
│ SharedMemoryWriter │
|
||||
└─────────────────────────────────────┘
|
||||
│
|
||||
│ memfd (fd passed via binder/property)
|
||||
↓
|
||||
Stage 2: Provider Process (HAL Injection)
|
||||
┌─────────────────────────────────────┐
|
||||
│ SharedMemoryReader │
|
||||
│ ↓ reads latest YUV frame │
|
||||
│ BufferConverter │
|
||||
│ ↓ handles format/stride conversion │
|
||||
│ AHardwareBufferWriter │
|
||||
│ ↓ lock → memcpy → unlock │
|
||||
│ HAL output buffer (to cameraserver) │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Buffer Lifecycle
|
||||
|
||||
1. **Allocation**: HAL allocates output buffers during `configureStreams()`. We observe and record buffer dimensions, format, and stride.
|
||||
|
||||
2. **Frame Production**: App process decodes video via MediaCodec, writes YUV frames to a ring buffer in shared memory (memfd). Each frame has a sequence number and timestamp.
|
||||
|
||||
3. **Frame Consumption**: In `processCaptureRequest()`, our hook:
|
||||
a. Reads the latest frame from shared memory
|
||||
b. Locks the HAL output buffer via `AHardwareBuffer_lock()` (for YUV_420_888)
|
||||
c. Copies frame data with stride conversion if needed
|
||||
d. Unlocks the buffer
|
||||
e. Returns to cameraserver (appears as real camera frame)
|
||||
|
||||
4. **Synchronization**:
|
||||
- Shared memory ring buffer uses atomic sequence numbers
|
||||
- Reader always grabs the latest complete frame (no blocking)
|
||||
- If no frame available, forward to real HAL (passthrough)
|
||||
|
||||
### Format Handling
|
||||
|
||||
| Stream Format | Strategy |
|
||||
|---|---|
|
||||
| YUV_420_888 (32) | Direct AHardwareBuffer_lock + memcpy |
|
||||
| IMPLEMENTATION_DEFINED (35) | Passthrough to real HAL (opaque GPU format) |
|
||||
| JPEG/BLOB (37) | Passthrough to real HAL (encode handled by HAL) |
|
||||
|
||||
### Memory Layout
|
||||
|
||||
Shared memory ring buffer (memfd):
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Header (256 bytes) │
|
||||
│ - magic: 0xCSWAPR00 │
|
||||
│ - frame_count: uint32 │
|
||||
│ - write_index: atomic<uint32> │
|
||||
│ - read_index: atomic<uint32> │
|
||||
│ - frame_width: uint32 │
|
||||
│ - frame_height: uint32 │
|
||||
│ - frame_size: uint32 │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Frame 0 (width × height × 3/2 bytes) │
|
||||
│ - Y plane: width × height │
|
||||
│ - U plane: width/2 × height/2 │
|
||||
│ - V plane: width/2 × height/2 │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Frame 1 ... │
|
||||
├─────────────────────────────────────────────┤
|
||||
│ Frame N-1 ... │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Ring size: 4 frames (enough for 30fps video with slight timing jitter).
|
||||
|
||||
### AHardwareBuffer Write Procedure
|
||||
|
||||
```cpp
|
||||
// In processCaptureRequest hook:
|
||||
for (int i = 0; i < request->output_buffer_count; i++) {
|
||||
auto& buf = request->output_buffers[i];
|
||||
auto stream = find_stream(buf.stream_id);
|
||||
|
||||
if (stream->format == FORMAT_YUV_420_888 && g_hook_state.virtual_camera_enabled) {
|
||||
// Lock buffer for CPU write
|
||||
AHardwareBuffer* ahb = AHardwareBuffer_fromNativeHandle(buf.handle);
|
||||
AHardwareBuffer_Desc desc;
|
||||
AHardwareBuffer_describe(ahb, &desc);
|
||||
|
||||
void* cpu_addr = nullptr;
|
||||
int result = AHardwareBuffer_lock(ahb,
|
||||
AHARDWAREBUFFER_USAGE_CPU_WRITE_OFTEN,
|
||||
-1, nullptr, &cpu_addr);
|
||||
|
||||
if (result == 0 && cpu_addr) {
|
||||
// Copy YUV frame from shared memory
|
||||
copy_yuv_frame_to_buffer(cpu_addr, desc.stride, desc.height,
|
||||
g_frame_reader->latest_frame());
|
||||
AHardwareBuffer_unlock(ahb, nullptr);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Tasks
|
||||
|
||||
- **Task 7**: Video decoder (MediaCodec NDK) in app process
|
||||
- **Task 9**: Buffer format conversion + shared memory ring buffer
|
||||
- **Task 6**: processCaptureRequest integration (AHardwareBuffer write)
|
||||
|
||||
## Pixel 9a Specifics
|
||||
|
||||
- API level: 36 (Android 16)
|
||||
- AHardwareBuffer: fully supported
|
||||
- Gralloc: `/vendor/lib64/hw/gralloc.gs101.so` (Tensor G3)
|
||||
- YUV_420_888: CPU-accessible, linear layout
|
||||
- IMPLEMENTATION_DEFINED: opaque, GPU-only (passthrough)
|
||||
Reference in New Issue
Block a user