Remote Data Engineer (Elasticsearch + Datawarehousing)

Description:

• Particle41 is seeking a talented and versatile Data Engineer to join their innovative team.
• The Data Engineer will play a key role in designing, building, and maintaining robust data pipelines and infrastructure to support clients' data needs.
• Responsibilities include working on end-to-end data solutions and collaborating with cross-functional teams to ensure high-quality, scalable, and efficient data delivery.
• The role offers an exciting opportunity to contribute to impactful projects, solve complex data challenges, and grow skills in a supportive and dynamic environment.
• Key responsibilities include designing, implementing, and optimizing Elasticsearch clusters for high-performance querying and data retrieval.
• The Data Engineer will build and manage Elasticsearch indexes, ensuring data is stored, indexed, and queried efficiently.
• They will also build and optimize data storage solutions like data lakes and warehouses.
• The role involves integrating structured and unstructured data from various internal and external systems to create a unified view for analysis.
• Ensuring data accuracy, consistency, and completeness through rigorous validation, cleansing, and transformation processes is essential.
• The Data Engineer will maintain comprehensive documentation for data processes, tools, and systems while promoting best practices for efficient workflows.
• Collaboration with product managers and other stakeholders to gather requirements and translate them into technical solutions is required.
• Participation in Agile development processes, including sprint planning, daily stand-ups, and sprint reviews, is expected.
• The role includes conducting thorough testing and debugging to ensure the reliability, security, and performance of applications.
• Continuous learning and innovation are encouraged, with a focus on staying updated with the latest technologies and trends in full-stack development.

Requirements:

• A Bachelor's degree in Computer Science, Engineering, or a related field is required.
• Proven experience as a Data Engineer, with a minimum of 3 years of experience is necessary.
• Proficiency in Elasticsearch and Python programming language is a must.
• Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases is required.
• A strong understanding of programming libraries/frameworks and technologies such as Flask, API frameworks, data warehousing/lakehouse principles, database ORM, data analysis, Databricks, Pandas, Spark, PySpark, machine learning, OpenCV, and Scikit-learn is essential.
• The ability to utilize Java to build and enhance backend systems, particularly for integration with Elasticsearch and databases, is needed.
• Experience in developing APIs, microservices, and automation scripts is required.
• Familiarity with utilities and tools such as logging, requests, subprocess, regex, and pytest is necessary.
• Knowledge of the ELK stack, Redis, and distributed task queues is important.
• A strong understanding of data warehousing/lakehousing principles and concurrent/parallel processing concepts is required.
• Familiarity with at least one cloud data engineering stack (Azure, AWS, or GCP) and the ability to quickly learn and adapt to new ETL/ELT tools across various cloud providers is necessary.
• Familiarity with version control systems like Git and collaborative development workflows is required.
• Competence in working on Linux OS and creating shell scripts is essential.
• A solid understanding of software engineering principles, design patterns, and best practices is necessary.
• Excellent problem-solving and analytical skills, with a keen attention to detail, are required.
• Effective communication skills, both written and verbal, and the ability to collaborate in a team environment are essential.
• Adaptability and willingness to learn new technologies and tools as needed are important.

Benefits:

• Particle41 values empowering leadership, innovation, teamwork, and excellence, driving everything they do to achieve the ultimate outcomes for clients.
• The company provides equal employment opportunities to all employees and applicants, ensuring that hiring and employment decisions are based on merit and qualifications.
• Particle41 welcomes individuals from all backgrounds who are committed to their mission and values.
• The company encourages applicants from diverse regions to apply and offers assistance during the application or interview process if needed.