- multi threading/multiprocessing
- rotating IPs (I will provide proxies)
- working with virtual machines (google compute engine)
- scraping using selenium (requests or urllib are very welcome if You can handle javaScript)
- working with database (expecting more than 200GB at end, speed is important)
Project is about scraping one website for page sources and inserting same into database. Since there are few million queries, I want to run it on virtual machine or even on few of them. Except You can handle javaScript with requests or urllib, than I can run it localy. Everything is about total time and getting at least 90% of results from website. Website is using a third-party anti web-scraping service Distil Networks (https://www.distilnetworks.com/). You must be able to handle it!
I already have code from previous developer but it is not working like I expected. You can modify this code or create Your own.
Write apple in first line when You apply so I know You read full post. Reply without this word will be ignored.
I need all buildings and their units from this page https://streeteasy.com/buildings/nyc There are four tabs for units, active listings , past sales, past rentals and all units. Tab all units doesnt have all units, so You must check in all tabs (each tab will require click() with selenium. otherwise data will not be inside page source)
I need four scripts:
1) collect_building_urls
- first script should collect all building urls from website and insert them into database without duplicates
2) collect_building_page_sources
- second script should collect page source for each building that is not already done and insert them into database
3) parse_unit_urls
- third script should parse each building page source that is not already done, find all unit links and insert them into database, without duplicates
4) collect_unit_page_sources
- fourth script should collect page source for each unit that is not already done and insert them into database
If You are interested for this project, Youll need to create four scripts, set them on virtual machine, test and make able to finish job.
Finding page sources and urls is very easy, few line of codes, but there are a lot of checking to make sure script is on right page and all content is loaded.
There must be log about each thread/process
I need this to be finished in next few days.
I already waste a lot of my time, so please apply only if You are able to finish this on time and let me know Your price for this project.
Conversation via skype will be required and also sharing screen if needed.","employmentType":["FULL_TIME","PART_TIME","CONTRACTOR","TEMPORARY","PER_DIEM"],"jobLocationType":"TELECOMMUTE","hiringOrganization":{"@type":"Organization","name":"Toogit","sameAs":"https://www.toogit.com/","logo":"https://www.toogit.com/images/toogit_logo_initial.png"},"identifier":{"@type":"PropertyValue","name":"Toogit","value":300062},"skills":["Multithreaded Programming","Python"],"applicantLocationRequirements":[{"@type":"Country","name":"IN"},{"@type":"Country","name":"Canada"},{"@type":"Country","name":"USA"},{"@type":"Country","name":"Germany"},{"@type":"Country","name":"Pakistan"},{"@type":"Country","name":"Philippines"},{"@type":"Country","name":"Indonesia"},{"@type":"Country","name":"Sri Lanka"},{"@type":"Country","name":"Nigeria"},{"@type":"Country","name":"China"},{"@type":"Country","name":"Russia"},{"@type":"Country","name":"Bangladesh"}],"validThrough":"2025-02-01T05:01:51+05:30","url":"https://www.toogit.com/freelance-jobs/MzAwMDYy"}
Remote Data Mining And Management Job In Data Science And Analytics
Find more Data Mining And Management remote jobs posted recently Worldwide
Work from Anywhere
40 hrs / weekHourly Type
Remote Job$143.73
Cost Looking for help? Checkout our video tutorial
How to search and apply for jobs
How to apply? Do you have more questions about the Job?
See frequently asked questions
NOTE: Make sure you know how to deal with aws sms/sns and aws login techniques with in python flask api.
We need one api with two methods will be describe it later. I will share some sample test cases and sample mysql stored procedure call s...read more
I have a d3 line chart with a maximum of 14 lines.
the full dataset of the chart is around 20millions of datapoints which i want to display.
therfore i need some solution to reduce the datapoints for visualisation in an performance and...read more
Background
We are working with a major university in Texas to demonstrate the power of giving and it impact on the entire worldwide academic network. The mission of the project is to create a storytelling platform that highlights the stories...read more
I want to solve a fractional differential equation. I already make neural network in TensorFlow but it has some bugs I couldnt fix it. May be you can do it.
We are looking for a full stack developer to add new features to an existing web application. Must have skills include 10+ experience with HTML, Python, and Java Script.
Preference given to candidates who have expertise with mongoDB, Atlas,...read more