Image55 multi turn dialogue latency
| Test Date | Total Test Cases | Concurrency | Prefix Cache Enabled | Chunk Acceleration Enabled | RAG Chunk Length | RAG Chunk Recall Count | RAG Chunk Hit Count | TopK Sparsification Enabled | Total Request Time (ms) | E2E Throughput (tokens/s) | Average Throughput per Request (tokens/s) | First Token Latency P50 (ms) | First Token Latency P90 (ms) | First Token Latency P99 (ms) | Max First Token Latency (ms) | Average First Token Latency (ms) | Incremental Latency P50 (ms) | Incremental Latency P90 (ms) | Incremental Latency P99 (ms) | Max Incremental Latency (ms) | Average Incremental Latency (ms) | Average Accuracy |
| 2025-10-16-14-31-31 | 279 | 1 | FALSE | FALSE | 3 | 1000 | 0 | FALSE | 465072.86 | 4.9 | 6.96 | 1151.86 | 2930.78 | 3429.74 | 3545.17 | 1452.4 | 41.47 | 69.7 | 105.57 | 11209.48 | 56.1 | 0.45091 |
| 2025-10-16-14-31-31 | 279 | 1 | TRUE | FALSE | 3 | 1000 | 0 | FALSE | 155829.15 | 14.63 | 17.13 | 217.37 | 686.87 | 995.54 | 1149.71 | 353.46 | 32.15 | 40.1 | 46.24 | 58.37 | 30.34 | 0.44326 |
| 2025-10-16-14-31-31 | 279 | 1 | TRUE | FALSE | 3 | 1000 | 0 | TRUE | 158653.1 | 14.36 | 16.79 | 218.61 | 700.11 | 981.06 | 1159.31 | 356.49 | 33.18 | 41.13 | 47.96 | 79.15 | 31.24 | 0.43334 |
| 2025-10-16-14-31-31 | 279 | 1 | TRUE | TRUE | 3 | 1000 | 0 | FALSE | 368143.72 | 6.19 | 6.71 | 1307.98 | 1570.18 | 1825.02 | 2107.98 | 1164.96 | 42.03 | 61.49 | 75.46 | 116.13 | 40.64 | 0.47634 |
| 2025-10-16-14-31-31 | 279 | 1 | TRUE | TRUE | 3 | 1000 | 3 | FALSE | 366629.04 | 6.22 | 6.74 | 1285.11 | 1581.82 | 1864.77 | 1898.59 | 1149.23 | 42.49 | 63.35 | 78.72 | 237.31 | 41.62 | 0.37617 |
| 2025-10-16-14-31-31 | 279 | 1 | TRUE | TRUE | 3 | 1000 | 0 | TRUE | 373218.55 | 6.11 | 6.62 | 1314.76 | 1601.37 | 1834.68 | 2127.12 | 1167.98 | 43.4 | 64.82 | 79.64 | 140.89 | 42.62 | 0.44979 |