Skip to content
All roles
Engineering · Model Gateway

Agent Infra Engineer (Model Gateway)

You’ll own the model access infrastructure supporting our agents and AI applications, with a focus on reliable calls to external model services and third-party model gateways. You’ll improve the reliability of multi-turn interactions, tool calls, and streaming output through protocol adaptation, channel routing, observability, and incident response, while balancing performance and cost.

What you’ll do

  1. 01Model integration and maintenance: integrate external model services and third-party gateways, handle integration testing, deployment, and ongoing maintenance, and track changes to model capabilities and APIs.
  2. 02Message and protocol adaptation: handle differences across models in message structures, request parameters, multi-turn context, tool calls, multimodal content, and streaming events to ensure compatibility for agent calls.
  3. 03Model gateway operations: deploy, configure, upgrade, and troubleshoot the model gateway; maintain channels, model mappings, access permissions, quotas, and usage settings.
  4. 04Call reliability: optimize timeouts, retries, rate limits, channel routing, and failover to reduce the impact of upstream service instability and cut unnecessary duplicate calls and costs.
  5. 05Monitoring and diagnosis: establish monitoring and alerts by model and channel; track call success rates, time to first token, response times, and stream interruption rates to detect and diagnose anomalies promptly.
  6. 06Validation and continuous improvement: improve API probes, compatibility regression tests, and load tests; work with business teams and upstream providers to resolve issues, drive postmortems, and implement improvements.

What we look for

  1. 01Experience with LLM API integration, model gateway maintenance, or related backend services; able to conduct API integration testing and diagnose and resolve production issues independently.
  2. 02Familiarity with message formats and API protocols for major models, including OpenAI Chat Completions / Responses, Anthropic Messages, and Gemini; understanding of message roles, content blocks, tool calls and result submission, streaming events, completion indicators, and usage statistics, with the ability to troubleshoot protocol conversion and message compatibility issues.
  3. 03Hands-on experience deploying and using New-API, with an understanding of request forwarding, channel selection, model mapping, and authentication, and the ability to maintain a production deployment independently.
  4. 04Familiarity with HTTP / HTTPS, JSON, SSE, and reverse proxies; able to identify whether issues originate in the application, gateway, network, or upstream service, and handle connection timeouts, proxy buffering, stream interruptions, authentication failures, and rate limiting.
  5. 05Familiarity with Linux, Docker, and logging and monitoring tools; proficiency in at least one programming language, such as Go or Python, with the ability to build tools for API probing, compatibility validation, and operations automation.
  6. 06Strong engineering practices and a sense of ownership, with the ability to document issues clearly, plan solutions, and follow through on fixes, validation, and documentation.

Nice to have

  • Proficiency with AI tools for reading code, debugging APIs, development, troubleshooting, and documentation to improve efficiency.
  • Experience extending New-API with channel adapters, protocol conversion, routing strategies, usage tracking, or monitoring capabilities.
  • Familiarity with Go and the ability to read New-API source code to identify and fix issues; relevant open-source contributions are a plus.
  • Experience with production agent environments, integrations across multiple model providers, high-concurrency gateways, or optimization of long-lived connections or model call costs.
Apply for this role