Data Modeler
Ventois is always looking for talented people to become its team members. We realize that productive people are always a good addition to our organization. So if you have the financial acumen and the passion to work with some of the biggest companies in the world, join in!
Project Location(s): Work location is Shrewsbury, MA with required travel to client locations throughout the USA.
Education: A minimum of a Bachelor’s degree in computer science, computer information systems, information technology, or a closely related field, or a combination of education and experience equating to the U.S. equivalent of a Bachelor’s degree in one of the aforementioned subjects.
Responsibilities
- Configure Spark streaming to get ongoing information from Kafka and store the stream information to HDFS.
- Collect data using Spark Streaming from AWS S3 bucket in near real time and perform necessary Transformation and Aggregations to build the data model and persist the data in HDFS.
- Involve in creating data pipelines and CI/CD deployment pipelines.
- Define and execute appropriate steps to validate various data feeds in Hadoop.
- Create partitions and buckets for the tables in the Hive data warehouse. Create topics for the respective tables from the source system.
- Involve in migrating the data from on-premises to AWS cloud platform.
- Optimize Hive QL/pig scripts by using execution engines like Tez, Spark.
- Coordinate with the business team to get the requirements and desired business outcome.
- Develop spark code using Scala and Spark SQL for processing data and faster testing.
- Implement Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
- Work closely with the team within Agile methodology.
Skills/Experience
- Great interpersonal communication skills;
- A keen eye for spotting data trends;
- Great analytical skills;
- A keen grasp of information technology;
- Professional demeanor;
- Personal accountability and strong work ethic;
- Professional, able to interact with vendors/clients;
- Positive, “can-do” attitude.