U.S. Bank Senior Big Data Spark Engineer - San Francisco, CA in SAN FRANCISCO, California
The Senior Big Data Spark Engineer position will be responsible for the analysis, design, testing, development and maintenance of best in class software experiences. The candidate is a self-motivated individual who can collaborate with a team and across the organization. The candidate takes responsibility of the software artifacts produced adhering to U.S. Bank standards in order to ensure minimal impact to the customer experience. The candidate will be adept with the agile software development lifecycle and DevOps principles.
• Design and build large scale data processing system (real-time and batch) to address growing AI/ML and Data needs of a Fortune 500 company
• Build a product to process large amount data/events for AI/ML and Data consumption
• Automate test coverage (90 %) for data pipelines. Best practices and frameworks for unit, functional and integration tests.
• Automate CI and deployment processes and best practices for the production data pipelines.
• Build AI/ML model based alert mechanism and anomaly detection system for the product. The goal is have a self-annealing product
Bachelor's degree, or equivalent work experience
5 years of relevant experience
• 6 to 8 years of overall experience in software development with 3 or more years of relevant experience in designing, developing, deploying and operating large data processing data pipelines at scale.
• 3 or more years’ experience with Apache Spark for Streaming and batch process
• Good knowledge on Apache Kafka
• Strong background in programming (Scala/Java)
• Experience on building reusable data frameworks/modules
• Experience on Airflow scheduler
• Experience with Containers, Kubernetes and scaling elastically
• Strong background in algorithms and data structures
• Strong analytical and problem solving skills
• Strong bent towards engineering solutions which increase productivity of data consumers
• Strong bent toward completely automated code deployment/testing (DevOps, CI/CD)
• Passion for data engineering and for enabling others by making their data easier to access.
• Some experience with working with and operating workflow or orchestration frameworks, including open source tools like Activiti, Spring Boot, Airflow and Luigi or commercial enterprise tools.
• Excellent communication (writing, conversation, presentation) skills, consensus builder
• Demonstrated ability to tackle tough coding challenges independently and work closely with others on a highly productive coding team.
R equired Skills Summary : Apache Spark, Apache Kafka, Scala/Java, NoSQL Databases, Elasticsearch & Kibana, Kubernetes, Docker Containers
Preferred Skills / Experience
• Knowledge of API Development
• Apache Flink experience
• Cloud experience
• DevOps skills
• Any other streaming technologies/tools experience
Job: Information Technology
Primary Location: CA-CA-San Francisco
Shift: 1st - Daytime
Average Hours Per Week: 40
Requisition ID: 200016984
U.S. Bank is an Equal Opportunity Employer committed to creating a diverse workforce.
U.S. Bank is an equal opportunity employer committed to creating a diverse workforce. We consider all qualified applicants without regard to race, religion, color, sex, national origin, age, sexual orientation, gender identity, disability or veteran status, among other factors.