Image48 benchmark static latency no ucm

Test Time Input Length Output Length Concurrency Prefix Cache Enabled Prefix Match Length Chunk Acceleration Enabled RAG Chunk Length RAG Chunk Recall Count RAG Chunk Hit Count TopK Enabled Total Request Time (ms) E2E Throughput (tokens/s) Average Throughput per Concurrency (tokens/s) First Token Latency P50 (ms) First Token Latency P90 (ms) First Token Latency P99 (ms) Max First Token Latency (ms) Average First Token Latency (ms) Incremental Latency P50 (ms) Incremental Latency P90 (ms) Incremental Latency P99 (ms) Max Incremental Latency (ms) Average Incremental Latency (ms)
2025-10-25-11-34-46 32000 1000 8 FALSE 0 FALSE 0 0 0 FALSE 90963.66 87.88 10.98 24422.24 38739.91 41966.26 42324.74 24444.57 46.94 57.53 67.32 37185.3 65.81
2025-10-25-11-34-46 32000 1000 10 FALSE 0 FALSE 0 0 0 FALSE 107658.52 92.85 9.28 29670.07 48286.5 52505.56 52974.34 29755.01 52.78 75.51 79.06 48232.73 76.95
2025-10-25-11-34-46 32000 1000 15 FALSE 0 FALSE 0 0 0 FALSE 146769.5 102.16 6.81 42651.82 71861.83 78398.78 79123.28 42776.58 64.78 92.42 98.69 75378.76 102.56
2025-10-25-11-34-46 32000 1000 16 FALSE 0 FALSE 0 0 0 FALSE 155417.23 102.87 6.43 45115.12 76402.52 83389.17 84161.73 45259.71 67.54 94.81 101.83 80565.44 108.43
2025-10-25-11-34-46 32000 1000 20 FALSE 0 FALSE 0 0 0 FALSE 242944.95 82.25 4.11 55483.47 194771.62 203483.22 204446.72 70537.73 99.89 102.22 120.82 85925.42 127.43
2025-10-25-11-34-46 32000 1000 25 FALSE 0 FALSE 0 0 0 FALSE 282621.5 88.32 3.53 68487.05 217090.74 228178.57 229404.91 100254.08 98.97 102.13 107.34 85631.21 118
2025-10-25-11-34-46 32000 1000 30 FALSE 0 FALSE 0 0 0 FALSE 323351.58 92.71 3.09 81344.56 241867.54 255455.68 256959.75 124975.91 98.97 101.78 112.41 85870.04 121.61
2025-10-25-11-34-46 32000 1000 32 FALSE 0 FALSE 0 0 0 FALSE 338025.93 94.59 2.96 86560.46 250849.01 265409.37 267030.6 133448.16 98.45 101.97 113 85738.12 124.03
2025-10-25-11-34-46 32000 1000 35 FALSE 0 FALSE 0 0 0 FALSE 418750.56 83.53 2.39 195084.85 265334.23 347202.79 382891.09 148929.94 100.14 102.33 128.57 86232.66 140.09
2025-10-25-11-34-46 32000 1000 40 FALSE 0 FALSE 0 0 0 FALSE 459477.7 86.94 2.17 207367.19 388357.44 406278.54 408249.33 179878.03 99.89 102.27 135.05 86148.48 130.29
2025-10-25-11-34-46 32000 1000 43 FALSE 0 FALSE 0 0 0 FALSE 484590.98 88.66 2.06 215570.74 403159.25 422771.25 424997.91 196975.35 99.84 102.23 141.78 86376.1 128.77
2025-10-25-11-34-46 32000 1000 45 FALSE 0 FALSE 0 0 0 FALSE 501470.08 89.69 1.99 221824.7 414146.9 434760.71 437101.81 208479.75 99.82 102.21 132.3 87114.06 129.03
2025-10-25-11-34-46 32000 1000 50 FALSE 0 FALSE 0 0 0 FALSE 539038.66 92.63 1.85 234014.44 435901.87 458892.33 461433.58 231974.87 99.68 102.12 132.55 86256.48 132.33
2025-10-25-11-34-46 32000 1000 55 FALSE 0 FALSE 0 0 0 FALSE 636669.98 86.34 1.57 247526.2 460124.04 585120.63 587825.25 262041.16 100.22 102.28 132.01 86379.8 136.64
2025-10-25-11-34-46 32000 1000 60 FALSE 0 FALSE 0 0 0 FALSE 677747.16 87.18 1.45 260371.44 583639.63 611007.71 614024.47 290528.32 100.17 102.22 132.33 86274.56 133.23
2025-10-25-11-34-46 32000 1000 64 FALSE 0 FALSE 0 0 0 FALSE 708085.94 89.89 1.4 270664.38 602145.09 631631.61 634888.36 311439.92 100.06 102.21 131.39 86255.76 133.51

Comments