OPTIMIZATIONS
FP16 Precision
BetterTransformer
Flash Attention 2
Batching (24x)
PAUSE
RESET
INFERENCE ENGINE
PRE-PROCESSING
POST-PROCESSING
TOKENIZER
DE-TOKENIZER
SYSTEM THROUGHPUT (TOKENS/SEC)
10 T/s