AI / DevOps Engineer ( Test Automation & Telemetry )
AI / DevOps Engineer (Test Automation & Telemetry) We are seeking a Senior AI / DevOps Engineer to work on advanced automated testing and telemetry systems supporting large-scale AI-driven workflows. You will operate inside a high-visibility engineering environment where automated agents interact with complex applications and generate telemetry events used to evaluate system behavior and reliability. This role requires engineers who are comfortable operating independently, collaborating with senior stakeholders, and working in evolving AI-driven infrastructure environments. What You’ll Be Working On The team builds and maintains automated testing infrastructure that:
- Drives automated application interactions
- Generates telemetry events and behavioral signals
- Monitors workflow success and system health
- Uses AI tooling to generate and analyze events
- Supports large-scale experimentation and evaluation
You will work closely with senior engineers and technical leads to expand event generation capabilities and build AI-assisted tooling for telemetry and automated evaluation pipelines. Core Responsibilities
- Build and maintain automated telemetry and event-generation systems
- Develop and improve AI-assisted tooling for event creation and monitoring
- Debug automated test runs and investigate anomalies in telemetry data
- Build alerts and monitoring for automated test systems
- Analyze workflow sequences and event chains to validate system behavior
- Work with internal engineering teams to improve test coverage and reliability
- Investigate dips or anomalies in telemetry dashboards and propose fixes
- Contribute to the evolution of agent-driven testing infrastructure
- Maintain consistent engineering visibility through written updates, documentation, and collaboration with partner teams.
- You will work on AI-assisted testing infrastructure and telemetry pipelines that generate and analyze large volumes of application events.
What Success Looks Like Within the first month(s), you will:
- Understand telemetry pipelines and automated testing workflows
- Analyze event sequences and system behavior using internal dashboards
- Build AI-assisted tooling to improve event generation efficiency
- Help scale event coverage toward large test suites
- Improve signal quality by reducing noisy or non-actionable alerts
Required Experience
- Strong software engineering experience (5+ years)
- Python or similar scripting language
- Linux development environments
- Debugging distributed or automated systems
- Experience working with telemetry, logging, or monitoring systems
- Familiarity with CI/CD or build systems
- Strong written communication and documentation skills
Strong Plus
- Experience with AI tooling or LLM-based workflows
- Experience building automation frameworks
- Experience with test infrastructure or QA automation
- Experience with mobile application testing (Android / iOS)
- Experience analyzing telemetry or event pipelines
- Experience working in large enterprise engineering environments
What We’re Looking For This role requires engineers who:
- Take ownership of systems, not just tasks
- Communicate clearly and proactively
- Work well in ambiguous environments
- Are comfortable collaborating with senior stakeholders
- Can debug complex systems without extensive hand-holding
- Move quickly while maintaining engineering discipline
Why This Role Is Unique You’ll be working at the intersection of:
- AI-driven automation
- large-scale telemetry systems
- agent-based evaluation infrastructure
This is a rare opportunity to work on the next generation of automated software evaluation systems used in complex AI environments.
About Us
Flowmentum builds high-impact engineering teams embedded within some of the world’s most advanced technology organizations. Our engineers work on DevOps infrastructure, automation systems, AI-driven workflows, and platform reliability. Location Remote (US-friendly time zones preferred) Engagement Contract / Consulting (long-term) Apply tot his job Apply To this Job