Workload Scheduler Administrator / Infrastructure Engineer

We are seeking a highly skilled (3-5+ years dedicated experience administering) IBM Workload Scheduler (IWS) Administrator to manage, maintain, and optimize our Partner's enterprise batch scheduling infrastructure. The successful candidate will be responsible for the end-to-end administration of the IWS environment hosted primarily on Red Hat Enterprise Linux (RHEL). This role requires a strong blend of IWS expertise, Linux system administration, and scripting to ensure high availability and seamless execution of critical business workloads.

Responsibilities

• Administer Production IBM Workload Scheduler (aka Tivoli Workload Scheduler) environment with 28,000 unique daily jobs across about 350,000 daily job runs, 44 servers, and three other change control environments
• Administer, install, configure, and patch/upgrade IWS components (Master Domain Manager, Dynamic Agents, Dynamic Pool, Dynamic Workload Console)
• Work with Product Owner on communicating work streams in Jira
• Manage job promotions with Workload Application Template-based process, ensuring safety checks for platform stability assessed on each job promotion
• Manage change control of four (4) separate change control environments, enforce change control standards and policies
• Maintain and continuously promote 99.17% Production platform uptime per calendar month (uptime calculated excluding planned change control outages, planned weekly maintenance windows) using SOP's, DevOps tools, and disciplined change control across change control environments
• Orderly communication of platform-impacting news to user community of about 500 developers and data engineers
• Production consists of 44 servers between MDM, DWC, and agents
• Resolve complex job failures, performance bottlenecks, agent issues, and infrastructure issues
• Advise on complex job scheduling design questions that less experienced scheduling support team
• Monitor the health of the scheduler environment, manage database maintenance, and perform backup/disaster recovery, and monthly failovers
• Define and maintain security policies, user authorizations, and authentication for the DWC
• Respond to Cybersecurity vulnerability assessments and PCI (and other regulatory) audit inquiries
• Design and implement Ansible automation and self-healing mechanisms to continuously reduce unplanned outages
• Coordinate with offshore performing SOP's during non-working hours
• Scripting Python with IWS REST API

• US Citizenship
• Strong experience with IBM Workload Scheduler architecture, especially Dynamic Workload Broker, V10.1+, high availability of MDM's managing Fault Tolerant Agent and Dynamic Agent agent architectures
• Strong conceptual understanding of Master Domain Manager (MDM), Backup MDM (BMDM), Dynamic Workload Console (DWC), Fault Tolerant Agent (FTA), Dynamic Agent (DA)
• Strong grasp of conman CLI to monitor and control production plan, check job/job stream/resource status
• Strong grasp of composer CLI to define, modify and extract scheduling objects
• Strong grasp of planman CLI to control pre-production plan and GUI mirroring
• Strong grasp of lifecycle of daily production planning process, phases of JNextplan/FINAL
• Proficiency in navigating the DWC web-based GUI to monitor workloads, manage user access security, and define scheduling objects
• Experience installing IWS components, applying Fix Packs, and Interim Fixes
• Troubleshooting with logs under TWSDATA/stdlist, adjusting trace level for netman, batchman, writer, mailman, etc.
• Strong experience with IBM WebSphere Liberty
• Strong grasp of reading messages.log, traces.log, FFDC logs
• Strong grasp of configuring JVM heap sizes
• Strong grasp of configuring tracing scope, tracing levels, tracing retention
• Strong experience with Red Hat Enterprise Linux 8+
• Deep familiarity with bash/shell commands for text processing (for example, grep, awk, sed), file manipulation, and system navigation
• Ability to manage, start, stop, and troubleshoot SystemD services using systemctl and journalctl for IWS agents and MDM
• Managing user accounts, groups, service accounts and deep knowledge of Linux file permissions (chmod, chown, ACL on local filesystems and NFS)
• Ability to monitor system performance using tools like top, htop, vmstat, iostat, and sar to troubleshoot bottlenecks and platform unresponsiveness
• Understanding of Logical Volume Manager (LVM) and filesystem usage
• Checking TCP port availability, firewall rules (firewalld/iptables), and connectivity between MDM and Dynamic Agents using netstat, ss, ping, curl, etc.
• Managing SSL/TLS certificates, private keystores, public truststores, and working with Certificate Authority
• Strong experience with scripting (Bash Shell, Python, etc.) for automation
• Understanding of networking principles
• Understanding of basic Oracle database administration, enough to troubleshoot with DBA's to prove when an issue is in Oracle
• Understanding of basic SQL to query job metadata
• Understanding of checking database connectivity
• Understanding of AWS cloud infrastructure
• Experience with using secrets manager (CyberArk PPM, Hashicorp Vault, or similar)