NLP, Python Code Optimization

We have a code base in Python that calculates an NLP match score by comparing project data with all the candidate profiles on our platform. This currently takes 50-60 seconds and is using up a lot of CPU/Memory (We use PostgreSQL on AWS EB2 instance). The results of NLP processing are perfect and we do not want to change that, just the speed of processing needs to decrease. Our master code base is in Ruby which exchanges data with Python through Rabbit MQ.

We are trying to:
1. Reduce the NLP processing time to LESS THAN 1 second, EVEN as the database of candidates scales.
2. Drastically reduce the cpu/memory usage while running the NLP analyses in Python

We are looking for someone to do the following to optimize NLP processing speed:

1. Refactor the code to remove most elements of it that are not needed

2. Optimize GUNICORN workers to work on matching jobs in parallel (Leading to faster processing and less memory)

3. Use Caching and Indexing of Candidate and Project Strings for faster processing

We need this job done in a week and do not think Steps 1 and 2 will take much time.
About the recuiter
Member since Mar 14, 2020
Winnersportswea
from Yamagata, Japan

---

---

Offer to work on this project closes in 123 days!
Are you interested in this Opportunity?

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions