Remote Intermediate Data Engineer - OP01505

Description:

• Join Dev.Pro, a US-based outsourcing software development company, as an Intermediate Data Engineer.
• Play a crucial role in a project that connects the art market with the digital-first financial ecosystem.
• Work with a global leader in artwork tokenization and blockchain-secured provenance.
• Engage in an exciting blockchain-related project that transforms the art market.
• Collaborate with world-class brands in a unique business domain.
• Contribute to building processes, tools, and products from scratch in a dynamic no-legacy startup environment.
• Take full responsibility for the data warehouse and pipeline, including planning, coding, reviews, and delivery to production.
• Migrate data from existing file storage systems to Google Cloud Platform, including Google Cloud Storage and BigQuery.
• Design, develop, and maintain ETL/ELT pipelines to support data migration and integration.
• Collaborate with team members to improve data cleaning and standardization capabilities.
• Conduct evaluations of the existing technology stack and provide data-driven recommendations for improvements.
• Develop a new scraper system to extract and aggregate data from diverse external sources.
• Ensure the integrity, consistency, and quality of data through optimized processes and validation protocols.
• Work closely with a small, dynamic team to meet project milestones effectively.

Requirements:

• A minimum of 4 years of experience in data engineering, including data extraction, transformation, and migration.
• Advanced experience with data extraction from unstructured files and legacy systems.
• Proven expertise in migrating data from file-based storage systems to cloud storage solutions, ideally on Google Cloud Platform.
• Proficiency with relational databases, specifically MariaDB or MySQL, and cloud-native solutions such as Google Cloud Storage and Google BigQuery.
• Strong programming skills in Python, focusing on data manipulation, automation, and re-implementing custom tools.
• Extensive experience with ETL/ELT pipeline development and workflow orchestration tools like Apache Airflow or Google Dataflow.
• Hands-on experience with batch processing frameworks and real-time data processing frameworks.
• In-depth understanding of data modeling, data warehousing, and best practices for designing scalable data architectures.
• Practical experience in developing or re-engineering data mastering tools for data cleaning and standardization.
• Expertise in RDBMS functionalities, including stored procedures, triggers, and indexes.
• Ability to handle Personally Identifiable Information (PII) data within pipelines and data storage systems.
• Experience with NoSQL databases such as MongoDB or Cassandra.
• Familiarity with monitoring tools like Prometheus and Grafana to oversee data pipelines.
• Knowledge of best practices in database management, performance optimization, and data security.
• Ability to critically evaluate data architecture and provide strategic recommendations for improvements.
• Upper-Intermediate+ English level.
• Desirable: Familiarity with JavaScript, experience with ElasticSearch, proficiency with analytical tools, interest in the art industry, and experience with PostgreSQL.

Benefits:

• Enjoy 30 paid rest days per year for holidays, vacation, or other personal time.
• Receive 5 sick leave days, up to 60 days of medical leave, and up to 6 days of leave for family reasons.
• Get a health insurance package fully compensated by Dev.Pro.
• Participate in fun online activities and team-building events.
• Benefit from continuous remote HR, payroll support, and overtime coverage.
• Join English/Polish lessons to enhance language skills.
• Grow your expertise with mentorship support and access to DP University for professional development.