If you can’t boat, dream…
As is almost everyone in the world right now, we are stuck and are having difficulty even imaging getting out on the boat in the foreseeable future. So to amuse myself I have been looking online for that future “forever” boat.
Who am I kidding; I am always looking online at boats. But what I have done is work on a project that fixes an annoyance of mine.
Some time in the recent past YachtWorld, which is the de facto standard for listing boats, decided to redo their website. And one of the outcomes of that is that you can no longer search for boats in multiple places at the same time. And seeing how I desire to buy a boat in the PNW (which used to be a selectable search category) and the PNW ostensibly includes British Columbia, Washington state and Oregon, I now had to perform three separate searches with no way to “save” a previous search and be able to compare. A definite downgrade if you ask me.
I had recently started to learn how to scrape websites using python because I wanted to be able to post a “books read” page on my personal site. Web scraping is a method of visiting a site and stripping the data from it. It occurred to me that I could apply the same procedure to YachtWorld and retrieve and aggregate all three searches to one output. So off I went.
Building
My initial method was to build a python script that output the data to a php web page. After a lot of trial and error it worked. But this meant every time I wanted to do a new search I would have to run the terminal command again and enter in the price range (which were the only variables I included). This produced a database file which the php page could then display. This was pretty onerous, particularly because I kept forgetting the proper commands :-). I also held out some hope that eventually I could share this with other people. There had to be a better way.
So I investigated further and discovered Flask, which is—simplistically put—a way to turn a python script into a web app. After a lot of fiddling (which I will detail soon over at macblaze.ca) I came up with this:
The new version added the options for Canadian vs. U.S. currency and length and the ability to sort the results. It is all one compact, self-contained unit and works 100% from a web browser. The search returns most of the details of the boats and links back to the official YachtWorld result so you can dive deeper into any particular boat. If you ask me it turned out pretty good.
A qualified success.
Deployment
I am stuck now though. It currently runs on my testing server and when I went looking to deploy it, discovered that my official web host doesn’t provide python support. I would either have to switch providers or get an additional (paid) account at another web host. I did find pythonanywhere.com, which offered free, albeit limited, python/flask hosting and was excited for the full hour it took to set up and get it running. You can find it here (http://btk.pythonanywhere.com/) but unfortunately the free accounts won’t let you scrape websites that don’t have official api’s (application programming interface)—which YachtWorld does, but it isn’t free so I don’t have access to it. So while the app is running it won’t actually deliver any results.
So now I am faced with the choice to either open up my testing server to the world (which I am not likely to do), change providers (and I just paid for the next couple of years) or sit on my little project and declare it for personal use only. I haven’t given up yet, but the prospects look dim*.
So…
So here I sit, a vast distance from the boat and any prospect of cruising and a weeks worth of work sitting on a computer with no way to share. And I am now getting tired of running the search over and over again. Sigh.
Anyway, if any of you experience the same frustration with YachtWorld, I sympathize and want you to know that there is a way around it with a bit of time and even less knowledge—which is something we should all have in abundance right now.
Update
*(May 2020) I managed to start a vm (virtual machine) to run the program. And I’ve added Sailboatlistings.com into the search as an option. The link (for now—it will likely get taken down at some point) is here: search.neverforever.ca. At this point I am still adding features but i assume I will eventually get bored or, if I am extremely luck, actually go cruising…
If you’re interested, this github repository has the whole flask app now.