Senior Splunk Engineer

Come join an amazing organization with stability, growth potential and an amazing work culture where you are valued.  We have more than 14,000 employees in 21 states and provide legendary service to 16 million loyal members.  Our membership continues to grow and we are always welcoming dedicated professionals looking to challenge themselves and build a career within our dynamic organization.    

The Automobile Club of Southern California is seeking an experienced Senior Splunk Engineer that can lead the design, architecture and development of monitoring solutions via integration of various toolsets into Splunk.  You will provide end to end monitoring solutions to all in-house hosted applications and be responsible for both platform administration and Splunk development activities.        

Requirements:

• Responsible for creating a comprehensive centralized monitoring system using Splunk as the base platform.        
• Create operational monitoring and dashboard solutions to read machine, infrastructure, application, web, third party services and other logs to monitor the health of infrastructure, applications, SLAs etc.        
• Experience gathering logs from various places, conform logs to standard structure for analysis, correlation and reporting.        
• Ability to properly define entities that support services being monitored in ITSI.        
• Experience using the ITSI module of Splunk to create glass tables and interdependencies within application maps.        
• Ability to define service, entity, glass table, and KPI dependencies for deep dive multi layered usage of ITSI.        
• Create an effective machine data-driven and market-informed log monitoring vision and strategy.        
• Create integration strategies and algorithms to ingest data into SPLUNK.        
• Onboarding new data as well as new applications to Spunk.        
• Build Multi KPI alerts to alert teams of degraded services or negative conditions.        
• Will work collaboratively with cross functional application teams to design dashboards and dashlets to expose problem areas.        
• Experience using JavaScript, Python or Ansible in writing scripts to manage monitoring of systems and dashboards.
• Mentoring other engineers within the organization on the capabilities and technical knowledge of the Splunk environment.        
• Write standard operating procedure documentation.        
• Prepare leadership and client presentations.        

• 2+ years' Splunk experience building dashboards and performing analytical searches within the Splunk application.        
• 2+ years' experience with Splunk ITSI modules and application mapping.   
• 2+ years' experience using or supporting centralized logging infrastructure -Splunk.        
• 2+ years' developing Splunk enterprise architecture, integration and deployment experience.        
• Experience implementing existing platforms and onboarding new applications.        
• Familiarity in writing scripts or applications in JavaScript, Python, or Ansible.     
• Familiarity with PERL/Linux shell scripting/Regex experience would be highly preferable.        
• A solid understanding of compute performance metrics and tools is highly desirable.        
• 2 years of hands-on experience in IT systems administration on Windows and UNIX platforms (Red Hat Linux, Solaris, AIX or equivalent)      
• General networking and security knowledge (firewalls, routing, DNS, NAT, packet trace and analysis, etc).      
• Splunk training and/or certifications would be a major plus.      

Benefits:

• Health Coverage for Medical, Dental, Vision
• Paid time off including Vacation, Illness and Holidays
• Life Insurance
• Disability Coverage
• Pension
• 401k Savings Plan
• Employee Discounts

'Creating members for life by exceeding our members' expectations through valuable products and legendary service'

AAA is an Equal Opportunity Employer