feat: nginx-based data parallel for optimal ASR throughput

When --dp N is specified (N > 1), the launcher now starts N independent
vLLM processes behind an nginx reverse proxy instead of using vLLM's
built-in DP coordinator. This avoids the single-process HTTP bottleneck
when handling large base64 audio payloads, achieving near-linear scaling
(7.2x with 8 GPUs at 4096 concurrent requests).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
Jianwei Yu
2026-03-27 07:43:32 +00:00
parent 9634518ca4
commit 3817f74d46
2 changed files with 275 additions and 22 deletions
+18 -1
View File
@@ -47,7 +47,9 @@ The launcher supports two types of GPU parallelism via `--tp` and `--dp` flags:
### Data Parallel (Recommended for scaling throughput)
Run 4 independent replicas on 4 GPUs — vLLM automatically distributes incoming requests:
Run 4 independent replicas on 4 GPUs with automatic load balancing behind a single port.
When `--dp N` is specified (N > 1), the launcher automatically starts N independent vLLM
processes behind an nginx reverse proxy for optimal throughput:
```bash
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
@@ -62,6 +64,21 @@ docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 4"
```
Run on all 8 GPUs:
```bash
docker run -d --gpus all --name vibevoice-vllm \
--ipc=host \
-p 8000:8000 \
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $(pwd):/app \
-w /app \
--entrypoint bash \
vllm/vllm-openai:v0.14.1 \
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 8"
```
### Tensor Parallel
Split a single model across 2 GPUs (useful if GPU memory is limited):