Open Telemetry (OTel) Instrumentation Engineer

3. Network Telemetry Data Collection: Gather network flow data (sFlow, NetFlow/IPFIX), SNMP traps, and BGP state via OTel-native and bridged receivers. Runtime Instrumentation: Instrument serverless (Lambda, Cloud Run) and container runtimes (EKS, GKE, AKS) utilizing auto-instrumentation where feasible, and manual instrumentation when technical nuance requires it.

Request ID: 103183-1
Title: Open Telemetry (OTel) Instrumentation Engineer
Location: Woodland Hills, CA (Hybrid)
Duration: 6 months
Pay Range: $40 - $47/Hour on W2/C2C (All inclusive)
 
Job Description:
 
Core Responsibilities
1. SaaS Instrumentation

• End-to-End Instrumentation: Instrument SaaS products utilizing OTel SDKs (Python, Node.js, Java, Go) to ensure spans, metrics, and structured logs are emitted at the appropriate granularity.
• Semantic Conventions: Define and enforce attribute naming conventions (e.g., service.name, tenant.id, feature.flag) aligned strictly with OTel semantic standards.
• Multi-Tenant Observability: Instrument multi-tenant surfaces to guarantee robust tenant-level observability while preventing cross-tenant data leakage.

2. Cloud Infrastructure

• Collector Fleet Management: Deploy and maintain the OTel Collector fleet across AWS, GCP, and Azure, including receiver configurations, processor pipelines, and exporter routing.
• Runtime Instrumentation: Instrument serverless (Lambda, Cloud Run) and container runtimes (EKS, GKE, AKS) utilizing auto-instrumentation where feasible, and manual instrumentation when technical nuance requires it.
• Metric Normalization: Collect and normalize cloud provider metrics (CloudWatch, Cloud Monitoring, Azure Monitor) via OTel receiver plugins.

3. Network Telemetry

• Data Collection: Gather network flow data (sFlow, NetFlow/IPFIX), SNMP traps, and BGP state via OTel-native and bridged receivers.
• Context Propagation: Correlate network events with application traces using consistent trace-context propagation across network boundaries.
• Strategy Collaboration: Partner with NetOps to define the MELT (Metrics, Events, Logs, Traces) strategy for on-prem, SD-WAN, and cloud interconnect segments.

4. Application Observability

• APM Ownership: Own Application Performance Monitoring (APM) instrumentation, including distributed tracing, custom span attributes, database query capture, and error fingerprinting.
• Asynchronous Workloads: Instrument async workloads (Kafka, SQS, Celery) ensuring W3C Trace Context propagation across message boundaries.
• SLI/SLO Definition: Define and instrument Service Level Indicators (SLIs) and Service Level Objectives (SLOs)—such as latency histograms, error-rate counters, and availability gauges—queryable directly from the Lakehouse.

5. Platform & Standards

• Internal Libraries: Build and maintain internal instrumentation libraries and language-specific wrappers that encode organizational conventions.
• Governance: Author and review instrumentation RFCs while driving the adoption of OTel semantic conventions across all engineering teams.
• Data Alignment: Collaborate with the AIOps ML team to ensure telemetry schemas meet feature engineering and ML model requirements.
• Pipeline Operations: Operate the Collector pipeline at scale, managing backpressure handling, sampling strategies (tail/head), and cardinality budgets.

Required Skills

• Strong expertise in:
  • OpenTelemetry (SDKs, APIs, Collector, OTLP)
  • Distributed tracing, metrics, and logging principles
• Experience with:
  • Instrumentation across Java, Python, Go, Node.js, and .NET
• Hands-on experience with Dynatrace

 
Experience Required:10+ Years
 
Company Benefits & Culture
• Opportunity to work with a dynamic team in a fast-paced environment
• Exposure to cutting-edge technologies and methodologies
• Supportive and collaborative work culture
 
Appreciate your quick response and please feel free to reach me out for any query you may have.
 
Thanks