Senior Splunk Engineer

AMERICAN SYSTEMS supports the mission of The US Space Force. Support includes assessing independent Local Area Networks (LANs); executing LAN collapse procedures; procuring authorized LAN equipment for network expansion or upgrades; conducting on-site surveys for new LAN locations; determining and documenting customer and technical requirements; transporting, configuring, and installing new equipment; and implementing all required services to make new locations fully operational networks.

AMERICAN SYSTEMS is seeking a professional with 8 – 10 years of experience and TS/SCI Clearance to be our next Senior Splunk Engineer at Malmstrom AFB, Montana.

Platform Engineering & Administration

• Install, configure, and maintain Splunk Enterprise and Splunk ES in classified, air-gapped, or cross-domain environments.
• Manage distributed architectures (indexers, search heads, cluster masters, deployment servers, forwarders) with a focus on reliability, performance, and security.
• Perform upgrades, patching, app deployment, performance tuning, and capacity planning.
• Implement and maintain backup/restore, DR procedures, and system hardening in accordance with DoD/IC and organizational policies.

Data Onboarding & Normalization

• Onboard logs from servers, network devices, security appliances, applications, and specialized classified systems.
• Develop and manage inputs, props, transforms, field extractions, and parsing to ensure high-quality, normalized data (CIM-compliant where applicable).
• Work with system owners and engineers to define logging requirements that support auditing, incident reconstruction, and compliance.
• Integrate Splunk with existing security tooling and infrastructure (e.g., host-based security, IDS/IPS, vulnerability scanners, identity systems).

Detection, Dashboards & Reporting

• Develop searches, correlation logic, alerts (where appropriate), and dashboards to surface security-relevant activity, system health, and compliance status.
• Create role-specific dashboards for cybersecurity staff, ISSOs/ISSMs, system administrators, and leadership.
• Support audit and inspection preparation (e.g., RMF, JSIG, NIST 800-53, CNSSI 1253) by building reports and evidence queries from Splunk.
• Implement and maintain data models, lookups, and other knowledge objects to support efficient analysis and reporting.

Security & Compliance Alignment

Ensure Splunk configurations and data flows comply with classified environment requirements, including handling caveats, data segregation, and need-to-know.

• Implement strict RBAC, data access controls, and logging of administrative actions.
• Support RMF and related processes by providing visibility into control effectiveness (e.g., AU-2, AU-6, AU-12, SI-4).
• Assist with continuous monitoring activities using Splunk as a key evidence and monitoring platform.

Collaboration & Technical Leadership (Non-SOC)

• Collaborate with cybersecurity engineers, ISSOs/ISSMs, system administrators, and network engineers to embed Splunk into system designs and modernization efforts.
• Provide expert guidance on how to leverage Splunk for troubleshooting, audit support, and security visibility.
• Mentor junior engineers and administrators on Splunk best practices, SPL queries, and platform usage.
• Contribute to Splunk standards, runbooks, and engineering documentation tailored for the classified environment.

Required Qualifications

• Active TS/SCI with CI Ploy clearance (or eligibility) as required by the program.
• Bachelor’s degree in Computer Science, Information Security, Information Systems, or equivalent experience.
• 8 - 10  years of experience with approximately 4-8 years of IT/cybersecurity experience, with at least 3+ years of hands-on Splunk Enterprise administration/engineering.
• Demonstrated experience supporting Splunk in highly regulated or secure environments (DoD, IC, federal, defense contractor, or similar).
• Proficiency with SPL, including complex searches, statistical commands, sub searches, lookups, and dashboard creation.

Experience onboarding and normalizing data from:

• Windows and Linux systems
• Network infrastructure (routers, switches, firewalls, proxies)
• Security tools (AV/EDR, IDS/IPS, vulnerability scanners, identity systems)
• Strong understanding of information security principles and controls (logging, monitoring, auditing, least privilege, configuration management).
• Familiarity with NIST 800-53, RMF, JSIG, or similar frameworks applicable to classified systems.

 Preferred Qualifications

• Splunk certifications (e.g., Splunk Core Certified Power User, Splunk Core Certified Admin, Splunk Enterprise Security Certified Admin).
• Experience operating Splunk in air-gapped, disconnected, or cross-domain (CDS) architectures.
• Scripting skills (Python, PowerShell, Bash) for automation, integrations, and data manipulation.
• Experience with configuration management and infrastructure-as-code (Ansible, Puppet, Chef, Terraform, or similar).
• DoD 8570/8140-compliant certification (e.g., Security+, CySA+, CASP+, CISSP, GSLC, GSEC) as required for IAT/IASAE roles.
• Background in one or more of: systems administration, network engineering, or cyber engineering in classified environments.

 Skills & Competencies

• Ability to work effectively in a classified, process-driven environment with strong attention to detail and documentation.
• Strong analytical and problem-solving skills; able to independently diagnose Splunk and data pipeline issues.
• Clear and concise communication skills for collaboration with technical teams and security leadership.
• Self-directed and able to prioritize tasks to support mission and compliance deadlines.