Senior Open Telemetry (OTel) Instrumentation Engineer

Artech LLC

Woodland Hills, CA, CA

JOB DETAILS
SALARY
$40–$48 Per Hour
SKILLS
AWS Lambda, Amazon Web Services (AWS), Apache Kafka, Application Programming Interface (API), BGP, Budgeting, Cloud Computing, Computer Security, Data Collection, Fleet Management, GCP (Good Clinical Practices), Instrumentation, Instrumentation Engineering, Java, JavaScript, Metrics, Microsoft .NET, Microsoft Windows Azure, Netflow, Node.js, Performance Analysis, Python Programming/Scripting Language, Receivers, SNMP (Simple Network Management Protocol), Simple Queue Service (SQS), Software as a Service (SaaS), Technical Writing, Telemetry, Vehicle Fleets, Web Client Plug-ins, World Wide Web Consortium (W3C), Writing Skills
LOCATION
Woodland Hills, CA, CA
POSTED
27 days ago
Request ID: 95626-1
Title: Senior Open Telemetry (OTel) Instrumentation Engineer

Locations: Onsite: Woodland Hills, CA
Duration: 6+ Months with Possible Extension.
Pay Range: $40 -$48/Hour on W2/C2C (All inclusive)

 
Job Descriptions:
 
Skills:
  • Dynatrace, Open Telemetry (SDKs, APIs, Collector, OTLP),Distributed tracing, metrics, logging principles, Instrumentation across languages (Java, Python, Go, Node.js, .NET)
  • Experience Required: 10 & Above
 
Core Responsibilities
1. SaaS Instrumentation
• End-to-End Instrumentation: Instrument SaaS products utilizing OTel SDKs (Python, Node.js, Java, Go) to ensure spans, metrics, and structured logs are emitted at the appropriate granularity.
• Semantic Conventions: Define and enforce attribute naming conventions (e.g., service.name, tenant.id, feature.flag) aligned strictly with OTel semantic standards.
• Multi-Tenant Observability: Instrument multi-tenant surfaces to guarantee robust tenant-level observability while preventing cross-tenant data leakage.


2. Cloud Infrastructure
• Collector Fleet Management: Deploy and maintain the OTel Collector fleet across AWS, GCP, and Azure, including receiver configurations, processor pipelines, and exporter routing.
• Runtime Instrumentation: Instrument serverless (Lambda, Cloud Run) and container runtimes (EKS, GKE, AKS) utilizing auto-instrumentation where feasible, and manual instrumentation when technical nuance requires it.
• Metric Normalization: Collect and normalize cloud provider metrics (CloudWatch, Cloud Monitoring, Azure Monitor) via OTel receiver plugins.


3. Network Telemetry
• Data Collection: Gather network flow data (sFlow, NetFlow/IPFIX), SNMP traps, and BGP state via OTel-native and bridged receivers.
• Context Propagation: Correlate network events with application traces using consistent trace-context propagation across network boundaries.
• Strategy Collaboration: Partner with NetOps to define the MELT (Metrics, Events, Logs, Traces) strategy for on-prem, SD-WAN, and cloud interconnect segments.


4. Application Observability
• APM Ownership: Own Application Performance Monitoring (APM) instrumentation, including distributed tracing, custom span attributes, database query capture, and error fingerprinting.
• Asynchronous Workloads: Instrument async workloads (Kafka, SQS, Celery) ensuring W3C Trace Context propagation across message boundaries.
• SLI/SLO Definition: Define and instrument Service Level Indicators (SLIs) and Service Level Objectives (SLOs)—such as latency histograms, error-rate counters*** and availability gauges—queryable directly from the Lakehouse.


5. Platform & Standards
• Internal Libraries: Build and maintain internal instrumentation libraries and language-specific wrappers that encode organizational conventions.
• Governance: Author and review instrumentation RFCs while driving the adoption of OTel semantic conventions across all engineering teams.
• Data Alignment: Collaborate with the AIOps ML team to ensure telemetry schemas meet feature engineering and ML model requirements.
• Pipeline Operations: Operate the Collector pipeline at scale, managing backpressure handling, sampling strategies (tail/head), and cardinality budgets."
• Strong expertise in: Open Telemetry (SDKs, APIs, Collector, OTLP),Distributed tracing, metrics, logging principles


Company Benefits & Culture
  • Inclusive and diverse work environment
  • Opportunities for professional growth and development
  • Comprehensive health and wellness benefits

Appreciate your quick response and please feel free to reach me out for any query you may have.

Thanks
 

About the Company

A

Artech LLC