Engineer/dev-ops experience writing Python code for data ingestion/processing

Looking for an experienced python developer to help with a data ingestion process. Needs:
- Data Engineering with Python 3
- Both Functional and OOP Algorithms
- Packaged with modules
- Advanced data structures
- Incorporate JSON control file
o how to validate/QA/cleanse erroneous or poor formatted incoming CSV files
o process & pipeline movement of data with compression, formats, local or remote destination, etc.
o error reporting (multiple file logging, email)
- Batchable (non-interactive) & interactive command-line driven
- CSV data cleaning/correcting
o Tracking all the cleaning operations performed on the data
o Tracking/reporting errors
- In-line/by-record Data transformations
- Familiarity of dev environments in Linux, git, and Linux/bash scripting
- AWS experience, such as S3 is an advantage
- AWS Lambda, Glue/Athena (Apache Hive) is an advantage
- multiprocessing in Python is an advantage
About the recuiter
Member since Nov 11, 2022
Jeevan Sethu
from New York, United States

---

---

Offer to work on this project closes in 20 days!
Are you interested in this Opportunity?

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions