feat: nginx-based data parallel for optimal ASR throughput
When --dp N is specified (N > 1), the launcher now starts N independent vLLM processes behind an nginx reverse proxy instead of using vLLM's built-in DP coordinator. This avoids the single-process HTTP bottleneck when handling large base64 audio payloads, achieving near-linear scaling (7.2x with 8 GPUs at 4096 concurrent requests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
@@ -47,7 +47,9 @@ The launcher supports two types of GPU parallelism via `--tp` and `--dp` flags:
|
||||
|
||||
### Data Parallel (Recommended for scaling throughput)
|
||||
|
||||
Run 4 independent replicas on 4 GPUs — vLLM automatically distributes incoming requests:
|
||||
Run 4 independent replicas on 4 GPUs with automatic load balancing behind a single port.
|
||||
When `--dp N` is specified (N > 1), the launcher automatically starts N independent vLLM
|
||||
processes behind an nginx reverse proxy for optimal throughput:
|
||||
|
||||
```bash
|
||||
docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
|
||||
@@ -62,6 +64,21 @@ docker run -d --gpus '"device=0,1,2,3"' --name vibevoice-vllm \
|
||||
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 4"
|
||||
```
|
||||
|
||||
Run on all 8 GPUs:
|
||||
|
||||
```bash
|
||||
docker run -d --gpus all --name vibevoice-vllm \
|
||||
--ipc=host \
|
||||
-p 8000:8000 \
|
||||
-e VIBEVOICE_FFMPEG_MAX_CONCURRENCY=64 \
|
||||
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
|
||||
-v $(pwd):/app \
|
||||
-w /app \
|
||||
--entrypoint bash \
|
||||
vllm/vllm-openai:v0.14.1 \
|
||||
-c "python3 /app/vllm_plugin/scripts/start_server.py --dp 8"
|
||||
```
|
||||
|
||||
### Tensor Parallel
|
||||
|
||||
Split a single model across 2 GPUs (useful if GPU memory is limited):
|
||||
|
||||
Reference in New Issue
Block a user