Job Description

Job Title: Data Engineer (Databricks, ETL, Data & AI Platforms)

Location: Remote in New York, Boston or Chicago

Job Type: Contract (40 hours a week)

We are working with a financial services firm that specializes in providing insights and analytics for banks, lenders, and other financial institutions. Their core focus is helping clients optimize their pricing, profitability, and customer engagement strategies through advanced data analysis and market intelligence

We are looking for a skilled and motivated Data Engineer with 3 years of expertise in Databricks, ETL processes, and building scalable data and AI platforms. This role is pivotal in supporting the migration of products and will involve designing, implementing, and optimizing data pipelines to ensure seamless data integration across various systems. The ideal candidate will be passionate about leveraging cutting-edge technology to build robust and efficient data systems to power business intelligence, analytics, and AI-driven initiatives.

Key Responsibilities:

• Databricks Development: Design, build, and optimize scalable data pipelines using Databricks to process large datasets and integrate various data sources into a unified data platform.
• ETL Pipelines: Develop and manage ETL processes to ingest, transform, and load data from diverse sources (on-premise and cloud-based) into data lakes or data warehouses. Ensure data quality, consistency, and integrity throughout the entire pipeline.
• Platform Development: Build and maintain data and AI platforms, ensuring that they are secure, efficient, and capable of handling high volumes of data. Collaborate closely with AI/ML teams to enable seamless integration of models into production systems.
• Product Migration Support: Assist in the migration of legacy systems and products to modern cloud-based data solutions. Ensure smooth data transfer, transformation, and system integration during the migration process.
• Data Modeling: Collaborate with data scientists, analysts, and business stakeholders to design and implement appropriate data models for business intelligence and machine learning workloads.
• Optimization and Monitoring: Optimize performance of data pipelines and platforms for speed and cost-effectiveness. Continuously monitor the health of data systems, troubleshoot issues, and implement improvements.
• Collaboration and Documentation: Work closely with cross-functional teams including data scientists, DevOps, and product managers to ensure alignment with business needs and best practices. Maintain comprehensive documentation for all data engineering processes, pipelines, and systems.
• Security and Compliance: Ensure all data solutions meet security, privacy, and compliance standards. Implement proper access controls and data governance measures.

Key Skills & Qualifications:

• Databricks: Proven experience working with Databricks, including developing and managing notebooks, jobs, and clusters, as well as leveraging Spark for distributed data processing.
• ETL Tools and Frameworks: Strong experience with ETL technologies (e.g., Apache Airflow, AWS Glue, or Azure Data Factory), and proficiency in building end-to-end data pipelines.
• Data Integration: Expertise in integrating structured and unstructured data from multiple sources, including relational databases, APIs, and cloud-based data sources.
• Cloud Technologies: Experience with cloud platforms such as AWS, Azure, or Google Cloud, including data storage services (e.g., S3, Blob Storage, BigQuery) and compute services.
• Data Warehousing: Experience with modern data warehousing solutions such as Snowflake, Redshift, or BigQuery.
• Programming Languages: Proficient in Python, Scala, or Java for building data pipelines, along with knowledge of SQL for querying and managing relational databases.
• AI/ML Collaboration: Familiarity with data science and machine learning concepts, with experience enabling AI workflows on data platforms.
• Problem-Solving: Strong troubleshooting, debugging, and performance optimization skills.
• Communication: Excellent communication and collaboration skills, able to work effectively with stakeholders at all levels of the organization.

Preferred Qualifications:

• Experience with automated deployment pipelines and CI/CD in data engineering workflows.
• Familiarity with data governance tools and frameworks (e.g., Apache Atlas, AWS Lake Formation).
• Experience with containerization technologies such as Docker and Kubernetes.
• Knowledge of data security principles and best practices in a cloud-based environment.

Job Tags

Similar Jobs