Lead Big Data Engineer

About the position The Lead Big Data Engineer, working independently, will develop, test, debug and document software components commensurate with their experience as well as direct development staff in support of a software engineering effort. We are seeking a Big Data Engineer to lead the design, develop, and maintain scalable data pipelines and infrastructure to support our market regulation technology applications. You will play a key role in modernizing existing data pipelines, enabling downstream users to experiment with data using no-code/low-code features. You will collaborate closely with product managers, data scientists, and engineers to deliver high-performance systems that power market surveillance and regulatory workflows. Responsibilities • Design and modernize scalable, modular data pipelines from legacy SQL-based workflows. • Build efficient and reliable ETL processes using Apache Spark and cloud-native tools on AWS. • Create configurable pipelines that accelerate experimentation and feature selection for data science teams. • Optimize pipeline performance, reduce storage costs, and improve compute efficiency. • Ensure compliance with governance standards while maintaining data integrity and lineage. • Integrate seamlessly with enterprise data platforms, lakes, and cloud-based storage systems. • Implement and maintain arenaflex/CD pipelines using Jenkins or similar tools. • Write robust automated tests, including unit, integration, and end-to-end coverage. • Collaborate within Agile teams, participating in sprint planning, reviews, and retrospectives. Requirements • Bachelor's degree in Computer Science, Information Systems or related discipline with at least seven (7) years of related experience, or equivalent training and/or work experience. • Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions. • Past experience with developing enterprise quality solutions in an iterative or Agile environment. • Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks. • Strong written and verbal technical communication skills. • Demonstrated ability to develop effective working relationships that improved the quality of work products. • Should be well organized, thorough, and able to handle competing priorities. • Ability to maintain focus and develop proficiency in new skills rapidly. • Ability to work in a fast paced environment. • Experience with object oriented programming languages such as Java, Scala or Python. Nice-to-haves • Experience in large-scale data engineering or data infrastructure roles. • Proficient in Python or Scala for building and maintaining data pipelines. • Deep expertise with Apache Spark, Hive, and distributed computing frameworks. • Strong knowledge of SQL and NoSQL databases (e.g., PostgreSQL, MongoDB, DynamoDB). • Experience with feature stores (e.g., Feast, Tecton, SageMaker Feature Store). • Hands-on with AWS services for compute, storage, and orchestration. • Experience implementing arenaflex/CD practices and managing production deployments. • Familiarity with Apache Airflow and workflow orchestration tools. • Experience with machine learning feature engineering and model serving frameworks. • Comfortable with automated testing and working in test-driven environments. • Proven ability to thrive in Agile/Scrum teams and contribute to iterative development cycles. Benefits • Comprehensive health, dental and vision insurance. • Basic life, accidental death and dismemberment, supplemental life, spouse/domestic partner and dependent life insurance. • Short- and long-term disability, long-term care, business travel accident, disability and legal insurance. • 401(k) plan with company match and eligibility for participation in an additional FINRA-funded retirement contribution. • Tuition reimbursement, commuter benefits, and other wellness benefits. • Generous time-off program of 15 days of paid time off, 5 personal days and 9 sick days. • Two volunteer service days. • Paid holidays. Apply tot his job

Back to blog

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...