Position Overview
The Observability Engineer builds and improves the capabilities used to understand the health, performance, and reliability of Goosehead applications and services. This hands-on role focuses primarily on application and service observability and partners with Cloud & Infrastructure on underlying infrastructure telemetry.
The engineer works with Software Engineering, DevOps, NOC, Incident Management, and Production Support to instrument services, improve telemetry, establish service-health and alerting standards, and accelerate production diagnosis. New Relic is a key platform, but the role is broader than administration of any single tool.
Core outcome: Make production health visible, alerts actionable, and failures easier to diagnose.
Key Responsibilities
- Design, implement, and improve observability for production applications, APIs, integrations, and critical services using OpenTelemetry for instrumenting metrics, logs, traces, and custom telemetry.
- Partner with Engineering to instrument applications using OpenTelemetry for structured logging, distributed tracing, and custom metrics.
- Create and maintain service-health dashboards, alerts, service maps, and diagnostic views that make availability, latency, errors, dependencies, and customer impact clear.
- Establish practical standards for instrumentation, monitoring coverage, alerting, dashboards, Service Level Indicators (SLIs), and Service Level Objectives (SLOs).
- Improve alert quality by reducing noise, false positives, duplicates, and non-actionable alerts; ensure alerts support effective NOC triage and escalation.
- Support Incident Management and Production Support during significant events by correlating telemetry, deployments, dependencies, and system behavior to speed diagnosis.
- Partner with Problem Management to identify observability gaps after recurring or significant incidents and implement preventive improvements.
- Partner with DevOps and Cloud & Infrastructure to connect application observability with CI/CD, deployments, cloud, infrastructure, and platform telemetry.
- Build scripts and automation using PowerShell, Python, Bash, APIs, or similar technologies to improve telemetry, configuration, alerting, and operational workflows.
- Help integrate observability and alert/event data with ITSM and incident workflows, including future ServiceNow capabilities.
- Maintain concise technical standards, reusable patterns, and operational guidance for Engineering and Technology Operations teams.
- Maintain service-based error budgets which our development teams use to dictate feature work.
Qualifications
- Minimum 3 years of hands-on experience in Observability Engineering, SRE, DevOps, Application Performance Monitoring, Production Engineering, Application Support Engineering, or a closely related role; 4-5 years preferred.
- Hands-on experience with application performance monitoring, metrics, centralized logging, distributed tracing, dashboards, alerting, and production troubleshooting.
- Experience instrumenting applications or services using OpenTelemetry for custom metrics, structured logs, and distributed tracing.
- Strong understanding of APIs, integrations, distributed applications, service dependencies, and common performance and reliability issues.
- Ability to correlate metrics, logs, traces, and change/deployment information to troubleshoot production issues.
- Hands-on scripting or automation experience using PowerShell, Python, Bash, APIs, or comparable technologies.
- Working knowledge of SLIs, SLOs, availability, latency, error rates, alert design, and service-health concepts.
- Working knowledge of cloud environments and the relationship between application and infrastructure telemetry.
- Strong troubleshooting, analytical, communication, and cross-team collaboration skills.
- Associate or bachelor's degree in Information Technology, Computer Science, Engineering, or related field preferred; equivalent technical experience considered.
Preferred Qualifications
- Hands-on experience with New Relic, including OpenTelemetry services, logs, dashboards, alerting, distributed tracing, or service maps.
- Experience with comparable observability platforms such as Datadog, Dynatrace, Splunk Observability, AppDynamics, Azure Monitor/Application Insights, Grafana, or Prometheus.
- Practical OpenTelemetry implementation experience and experience defining reusable observability or instrumentation standards.
- Experience establishing SLIs/SLOs or service-health measures for production applications.
- Experience integrating observability platforms with incident, event, notification, or ITSM workflows; ServiceNow experience is a plus.
- Experience supporting Microsoft Azure or other public cloud environments and integrating observability with CI/CD or deployment processes.
- Familiarity with ITIL Incident Management and Problem Management practices.