Image46 benchmark static latency

Test Time Input Length Output Length Concurrency Prefix Cache Enabled Prefix Match Length Chunk Acceleration Enabled RAG Chunk Length RAG Chunk Recall Count RAG Chunk Hit Count TopK Enabled Total Request Time (ms) E2E Throughput (tokens/s) Average Throughput per Concurrency (tokens/s) First Token Latency P50 (ms) First Token Latency P90 (ms) First Token Latency P99 (ms) Max First Token Latency (ms) Average First Token Latency (ms) Incremental Latency P50 (ms) Incremental Latency P90 (ms) Incremental Latency P99 (ms) Max Incremental Latency (ms) Average Incremental Latency (ms)
2025-10-22-17-44-05 32,000 1,000 8 TRUE 32,000 FALSE 0 0 0 FALSE 85,780.91 93.26 11.66 18,518.41 25,439.70 25,571.30 25,585.93 18,165.36 57.52 74.79 110.46 6,217.31 63.91
2025-10-22-17-44-05 32,000 1,000 10 TRUE 32,000 FALSE 0 0 0 FALSE 96,533.36 103.59 10.36 28,079.56 29,884.91 31,350.21 31,513.03 26,174.26 60.6 79.68 108.47 10,679.71 68.74
2025-10-22-17-44-05 32,000 1,000 15 TRUE 32,000 FALSE 0 0 0 FALSE 119,226.68 125.81 8.39 40,189.59 45,103.22 46,839.84 46,923.57 38,726.93 67.43 90.69 128.95 13,746.20 77.44
2025-10-22-17-44-05 32,000 1,000 16 TRUE 32,000 FALSE 0 0 0 FALSE 123,232.60 129.84 8.12 41,632.13 47,061.69 49,218.10 49,344.99 38,901.22 68.36 93.56 133.27 25,462.87 80.98
2025-10-22-17-44-05 32,000 1,000 20 TRUE 32,000 FALSE 0 0 0 FALSE 140,231.46 142.62 7.13 55,348.72 57,472.77 60,741.38 61,075.68 52,450.65 71.55 101.73 145.37 27,604.10 84.75
2025-10-22-17-44-05 32,000 1,000 25 TRUE 32,000 FALSE 0 0 0 FALSE 165,283.94 151.25 6.05 61,958.80 74,438.92 78,908.76 79,036.35 59,713.67 80.07 109.4 229.68 45,309.60 99.8
2025-10-22-17-44-05 32,000 1,000 30 TRUE 32,000 FALSE 0 0 0 FALSE 184,653.46 162.47 5.42 64,182.60 89,682.59 93,682.55 93,784.61 65,833.11 86.25 123.56 469.23 50,568.58 109.6
2025-10-22-17-44-05 32,000 1,000 32 TRUE 32,000 FALSE 0 0 0 FALSE 215,697.57 148.36 4.64 65,592.70 96,443.42 100,625.76 100,728.75 70,790.73 103.1 141.4 470.08 43,330.75 120.65
2025-10-22-17-44-05 32,000 1,000 35 TRUE 32,000 FALSE 0 0 0 FALSE 278,088.20 125.86 3.6 63,910.60 98,771.03 218,409.53 218,723.94 81,115.75 113.62 154.04 581.4 43,352.84 138.79
2025-10-22-11-38-33 32000 1000 40 TRUE 32000 FALSE 0 0 0 FALSE 284151.92 140.77 3.52 69628.14 215766.24 223676.83 223818.89 97355.38 101.73 145.82 466.63 52113.75 123.35
2025-10-22-11-38-33 32000 1000 43 TRUE 32000 FALSE 0 0 0 FALSE 299622.53 143.51 3.34 74925.12 226210.16 234017.41 234098.26 108931.65 99.7 143.98 520.28 40761.12 120.64
2025-10-22-11-38-33 32000 1000 45 TRUE 32000 FALSE 0 0 0 FALSE 311883.43 144.28 3.21 80069.47 235644.52 243442.13 243551.69 117428.31 100.39 143.28 480.32 53699.98 121.21
2025-10-22-16-19-32 32,000 1,000 50 TRUE 32,000 FALSE 0 0 0 FALSE 337,093.51 148.33 2.97 89,648.59 253,492.73 261,249.71 261,399.11 132,717.75 100.45 143.43 465.81 57,313.16 123.41
2025-10-22-16-19-32 32,000 1,000 55 TRUE 32,000 FALSE 0 0 0 FALSE 360,813.06 152.43 2.77 95,221.04 266,286.92 273,553.80 273,723.05 147,437.36 101.48 143.73 513.66 41,118.85 135.92
2025-10-22-16-19-32 32,000 1,000 60 TRUE 32,000 FALSE 0 0 0 FALSE 392,466.46 152.88 2.55 104,485.99 276,576.81 295,786.50 296,707.94 162,241.61 102.27 147.56 710.75 57,328.06 144.41
2025-10-22-16-19-32 32,000 1,000 64 TRUE 32,000 FALSE 0 0 0 FALSE 450,716.20 142 2.22 165,433.12 290,386.98 323,477.35 346,011.54 173,290.73 111 165.44 854.34 57,468.75 155.16

Comments