Solution Architect - Observability

About Blackhawk Network

Today, through BHN’s single global platform, businesses of all kinds can tap into the world’s largest network of branded payment solutions. BHN helps businesses grow revenue, increase loyalty, motivate and reward their teams, disburse funds and engage consumers. Branded payment solutions include the issuance and distribution of gift cards, egifts, corporate payouts and rewards, along with the technology to deliver these products in seamless, integrated ways. BHN’s network spans the globe with more than 400,000 consumer touchpoints. Learn more at BHN.com.

Hybrid flexibility: At Blackhawk Network, you’ll enjoy the best of both worlds—focused remote work plus in-person collaboration on Tuesdays and Wednesdays, our regular in-office days at our Pleasanton headquarters. This rhythm gives you the tools, connection, and autonomy you need to make a real impact.

Overview

Blackhawk Network is seeking an exceptional Solution Architect- Observability to design and scale our enterprise observability capabilities. This role combines deep technical expertise in modern observability platforms with a strong architectural mindset, enabling you to define how observability is implemented across our systems and engineering practices.

This role is a true observability solution architect position, where you will define how observability is designed, implemented, and scaled across the organization. You will not only select and configure tools but also shape the architecture, standards, and design patterns that guide how systems are instrumented and monitored. You will influence how hundreds of engineers approach system reliability and performance through thoughtful architecture and reusable solutions. 

Responsibilities

Observability Architecture and Strategy

• Define and drive the overall observability architecture vision aligned with business and platform goals
• Design end-to-end observability solutions across microservices, distributed systems, and AWS cloud environments
• Architect scalable telemetry pipelines for metrics, logs, traces, and events using tools such as New Relic, Splunk, Coralogix, and CloudWatch
• Establish reference architectures, design patterns, and standards to ensure consistent implementation across teams
• Ensure observability solutions are scalable, cost-efficient, and compliant with security and regulatory requirements
• Integrate observability into platform services, internal developer platforms, and CI/CD pipelines

Solution Design and Engineering Enablement

• Partner with engineering, product, and SRE teams to design observability solutions aligned with application and domain needs
• Act as a solution architect and consultant, guiding teams on instrumentation strategies and monitoring design
• Translate business and operational requirements into scalable observability architectures
• Create reusable architecture artifacts, documentation, and implementation blueprints
• Lead design reviews and provide hands-on guidance for complex observability challenges
• Enable teams with self-service observability capabilities while maintaining governance and standards

Technical Leadership and Governance

• Define and enforce observability standards, governance models, and best practices across the organization
• Drive improvements in mean time to detection and mean time to resolution through architectural improvements
• Evaluate emerging tools and technologies to evolve observability capabilities
• Collaborate across platform, SRE, and DevOps teams to embed observability into the software development lifecycle
• Mentor engineers and influence system design decisions across multiple teams and domains

Qualifications

Required Experience and Skills

• Bachelor’s degree in Computer Science, Information Systems, or related field (or equivalent experience)
• 6+ years of experience in platform engineering, SRE, DevOps, or solution architecture roles
• Proven experience designing and implementing large-scale observability architectures in production environments
• Expertise with modern observability platforms such as New Relic, Splunk, Datadog, or Coralogix
• Strong understanding of distributed systems and cloud-native architectures, preferably in AWS
• Hands-on experience with observability pipelines and frameworks such as OpenTelemetry, Fluentd, Vector, or Logstash
• Experience integrating observability into CI/CD pipelines and developer workflows
• Strong communication skills with the ability to influence and guide technical and non-technical stakeholders

Technical Skills

• Deep understanding of observability pillars: metrics, logs, traces, and events
• Experience designing instrumentation strategies using APM agents and OpenTelemetry
• Familiarity with query languages such as PromQL, SPL, NRQL, or LogQL
• Expertise in building dashboards and visualization strategies for operational insights

Benefits

Salary Range for California Residents Only: $138,140.00 to $186,500.00

Pay is based on several factors including but not limited to education, work experience, certifications, etc. In addition to your salary, Blackhawk Network offers benefits including 401k with employer match, medical, dental, vision, 12 paid holidays in the year 2026, 1 hour of sick pay accrual for every 30 hours worked, parental leave, life insurance, disability insurance, accident and illness insurance, health and dependent care flexible spending accounts, wellness benefits, and flexible time off for all full-time employees. 

EEO Statement

Blackhawk Network provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.  Blackhawk Network believes that diversity leads to strength. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

Blackhawk Network encourages applicants with previous criminal records to apply to all positions and, pursuant to the San Francisco and Los Angeles Fair Chance Acts (and other “Fair Chance” laws), Blackhawk Network will consider for employment qualified applicants with arrest and conviction records.  For Philadelphia applicants or jobs, please see a copy of Philadelphia’s ordinance on this topic by clicking this link: https://codelibrary.amlegal.com/codes/philadelphia/latest/philadelphia_pa/0-0-0-280104.