Job Description
Position:- Databricks Engineer ,
Location:- O Fallon, MO / Seattle - WA – Onsite ,
Client :- TCS ,
Duration:- Full Time Position,
Job Description:-
Experience Required - 8+ Years
Must Have Technical/Functional Skills
- Tenured in the fields of Computer Science/Engineering or Software Engineering.
- Bachelor's degree in computer science, or a related technical field including programming with 15+ years of demonstrated experience in software development and delivery.
- Deep hands-on experience with Databricks and Python.
- Spark and SQL Expertise: Writing performant Spark SQL and PySpark transformations. Optimizing joins, window functions, and aggregations. Handling large-scale data with partitioning and caching strategies
- Experience with real-time data processing and streaming pipelines.
- Proficient in both Delta Live Tables (DLT) and Workflows – Understanding of Lakeflow Declarative Pipelines is an asset.
- DevOps and CI/CD - Using Databricks Asset Bundles (DABs) for deployment - Integrating with Git, Bitbucket, and Jenkins for version control and automation
- Proven track record using dbt for building robust, testable data transformation workflows following TDD.
- Strong grasp of cloud-based database infrastructure (AWS, Azure, or GCP).
- Skilled in developing insightful dashboards and scalable data models using Power BI.
- Expert in SQL development and performance optimization.
- Demonstrated success in building and maintaining data observability tools and frameworks.
- Proven ability to plan and execute deployments, upgrades, and migrations with minimal disruption to operations.
Roles & Responsibilities: -
- Focus on standing up/adopting tools that will enable both internal and external customers. Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
- Work efficiently within a high security, PII and PCI-DSS environments
- Key leader/contributor in the data and software and platform and data warehouse development life cycle (i.e. design, development, documentation, testing, deployment, support)
- Support architects and engineers as they design and build effective, agile applications and platforms
- Design, develop, and optimize batch and real-time data pipelines using Medallion Architecture, preferably on Databricks cloud data platforms
- Build data transformation workflows using dbt, with a strong focus on Test-Driven Development (TDD) and modular design. < /p>
- Implement and manage CI/CD pipelines using GitLab and Jenkins, enabling automated testing, deployment, and monitoring of data workflows.
- Drive DataOps practices by integrating testing, monitoring, versioning, and collaboration into every phase of the data pipeline lifecycle.
- Build scalable and reusable data models that support business analytics and dashboarding in Power BI.
- Develop and support real-time data streaming pipelines (e.g., using Kafka, Spark Structured Streaming) for near-instant data availability.
- Establish and implement data observability practices, including monitoring data quality, freshness, lineage, and anomaly detection across the platform.
- Plan and own deployments, migrations, and upgrades across data platforms and pipelines to minimize service impacts, including developing and executing mitigation plans.
- Collaborate with stakeholders to understand data requirements and deliver reliable, high-impact data solutions.
- Document pipeline architecture, processes, and standards, promoting consistency and transparency across the team.
- Apply exceptional problem-solving and analytical skills to troubleshoot complex data and system issues.
- Demonstrate excellent written and verbal communication skills when collaborating across technical and non-technical teams.
Job Tags
Full time,