Subletting in Waterloo

I'm currently a student at the University of Waterloo, and this summer I was faced with the same issue that has, at one point, plagued every student at the University of Waterloo: finding a 4-month fall sublet. Nobody wants to commit to a full year because of the coop program, all the new students make the leasing and subletting market even tighter; resulting in it being almost impossible to find a 4-month sublet. The market for sublets, which exists almost entirely as posts on a Facebook group, is competitive to the point that if you see a post for someone selling their place for 4 months in the fall, and it's more than 15 minutes ago, don't bother messaging them. It's already taken. So how, besides sheer luck, am I to find a place to stay for 4 months? I decided to apply data analytics method I've been utilizing for years: creating a Facebook shell app to acquire and analyze posts on a page.

Getting Data From Facebook

Facebook's Graph API is clunky. It's hard to find what you're looking for, especially when that happens to be something it's not intended to be used for. The kind of unrestricted data that includes all publicly made posts that SRM companies sell to other companies like McDonald's to find people who have posted about them, is NOT, under ANY circumstances, available to regular users. The kind of data available to users, in an unintended way, is posts made on groups and pages that you, the individual user, have access to. Here's how to get it:

• Identify The Group ID — This is usually easy to find, generally it is in the link of the group.
• Create a "Shell App" — Go to developers.facebook.com and follow the directions.
• Get a Regular User Token — Go to the Graph API explorer and get a User access token.
• Get Extended User Token — Use this URL and click "get extended token." Should last about a month.

After a while your app gains the ability to extend the user token in perpetuity. Unfortunately, Facebook changes their API versions frequently, so watch out for deprecated functionality. I'm using version 2.10 as of this write up.

Getting the data is as easy as making a URL request. For a more complete list of the optional arguments, look at the Facebook Graph API documentation.

Processing The Data

Using the method above to get data from the housing post, I implemented different techniques to guess what kind of posts they were.

• Name, Date, and Post Message — These are acquired from the post itself.
• Term — The school term, easily calculated from date.
• Gender — Is this a female only unit? Use RegEx's to look for hook terms like "Female Only."
• Period, Location, Price — All calculated by looking at a series of RegEx's.
• Action — Whether the post is buying or selling. I tried 3 methods: NLTK/sklearn (40-57%), Random Forest on RegEx matrix (55-60%), and pure RegEx voting (75-80%). I used the last method.

I put the data in a SQLite database. I fetched data between every 5-18 seconds, checking for new posts, and categorized them. The code ran for an hour, then, using a Raspberry Pi, I executed it each hour from 5AM to 2AM using a cron job.

Twilio

The problem I have with the Waterloo housing Facebook group is that only 2.7% of the posts are students selling their place for 4 months in the fall. However, using the data analysis method described above, I am able to identify approximately 77% of the 4-month sublets posted within ~5 seconds of them being posted. Twilio allows you to buy a phone number and use it programmatically through Python. I set up a free account and hooked it up to my Python method. After setting it up on my Raspberry Pi, I was able to receive a text message to my phone, with the link to the post and its details, ~10 seconds after someone posted. This gave me a huge advantage on the best places to live.

VIEW CODE ON GITHUB