Lead Data Engineer for AI Social Media Platform / Remote

Are you passionate about building cutting-edge data infrastructure that powers AI-driven social media solutions? Our rapidly growing platform—dedicated to revolutionizing content creation through artificial intelligence—seeks an experienced Lead Data Engineer to spearhead our data architecture initiatives. This remote position offers an opportunity to work at the fascinating intersection of big data, machine learning, and social media analytics.

Key Responsibilities

- Architect and implement enterprise-grade data pipelines for collecting, processing, and storing data from diverse social media sources and user interactions at scale.

- Design and develop modern data warehouse solutions utilizing cloud-native technologies (Snowflake, BigQuery, Redshift) to support AI-driven content generation and analytics.

- Establish rigorous data quality frameworks and validation processes to ensure integrity, accuracy, and reliability of social media datasets powering our machine learning models.

- Automate Extract, Transform, Load (ETL) processes using industry-standard tools like Apache Airflow, Dagster, or dbt to streamline data ingestion with minimal manual intervention.

- Monitor performance metrics and continuously optimize data infrastructure for improved throughput, reduced latency, and enhanced scalability as our platform grows.

- Collaborate closely with Data Scientists and ML Engineers to deliver high-quality datasets for model training, ensuring our AI assistants maintain superior performance.

- Implement comprehensive data governance frameworks ensuring compliance with GDPR, CCPA, and other privacy regulations across all data handling processes.

- Establish robust monitoring solutions with alerting capabilities to proactively identify and address anomalies in data pipelines before they impact downstream systems.

- Partner with business stakeholders to design interactive dashboards and data marts providing actionable insights into social media trends and platform performance.

- Research and evaluate emerging technologies in the data engineering landscape, recommending strategic adoptions to improve our data architecture.

Required Qualifications

- Bachelor's or Master's degree in Computer Science, Data Engineering, or related technical field.

- 5+ years of professional experience in data engineering roles, with at least 2 years in a leadership position.

- Strong proficiency in Python 3.10+ and SQL, with demonstrable experience building production-grade data pipelines.

- Extensive experience with cloud data services (AWS Redshift/Glue/S3, Azure Synapse, Google BigQuery) and infrastructure-as-code tools.

- Hands-on expertise with modern ETL/ELT frameworks such as Apache Airflow, Dagster, or dbt.

- Deep understanding of data modeling techniques and experience designing dimensional schemas.

- Experience implementing data quality monitoring and testing frameworks within automated pipelines.

- Proficiency with version control systems (Git) and CI/CD practices for data infrastructure.

- Proven track record of optimizing data systems for performance, reliability, and cost-efficiency.

- Knowledge of data security best practices and experience implementing data access controls.

- Strong communication skills with ability to translate complex technical concepts to non-technical stakeholders.

Nice to Have

- Experience with real-time data streaming technologies (Apache Kafka, AWS Kinesis, Pulsar).

- Familiarity with data lake technologies (Delta Lake, Apache Iceberg, Hudi).

- Knowledge of container orchestration platforms like Kubernetes for data workloads.

- Experience with data observability tools (Monte Carlo, Datadog, Prometheus).

- Understanding of machine learning workflows and MLOps principles.

- Previous work with social media APIs and unstructured data processing.

- Contributions to open-source data engineering projects.

- Experience working in a fast-paced startup environment.

Why Join Us

Join our innovative team and shape the future of AI-powered social media content creation. You'll work with cutting-edge technologies in a remote-first environment that values work-life balance. We offer competitive compensation, continuous learning opportunities, and the chance to solve complex data challenges that impact thousands of businesses worldwide. As a key member of our engineering team, you'll have significant input into our technical direction and data strategy as we scale.