Autoscaling endpoints for LLM inference
Capability change
Together AI announces autoscaling endpoints for LLM inference on its Dedicated Model Inference platform, allowing users to configure scaling based on metrics like in-flight requests, TTFT, GPU utilization, and token throughput, with adjustable replica bounds and timing windows.
