Summer's Over

2012-08-27T00:00:00+00:00

Now that summer is over and school is back in session, I’ll be posting less and working more on schoolwork. I’m learning C++, C, MIPS Assembly, and Verilog in courses this semester and a lot more outside of class.

Working at SimpleRelevance was awesome and was a great experience! I’m going to miss the team and 1871, but I hope to be back at some point - but if not, it was everything I could have asked for, and more.

Celery, Redis, and RabbitMQ

2012-07-15T00:00:00+00:00

This summer I’ve been developing and hacking away at SimpleRelevance, in “Research and Development”. SimpleRelevance is a recommendation system that’s really easy to set up and use. At it’s core, it pairs customers with products or services they’d like, and makes it easy for online stores or marketers to use that data.

This kind of personalization involves plenty of calculations and mathemagical calculations. We use Celery to distribute these tasks to several worker boxes.

We’ve used RabbitMQ in the past to queue up these tasks and serve them to the Celery “workers”, which then returned the results.

However, we decided to switch to Redis after reading all about it and how it would speed up our infrastructure immensely.

RabbitMQ is huge, takes up a lot of RAM, takes forever to push tasks (more on this later), and is just…well, big. Redis is a simpler system that integrates caching and is extremely fast and much more lightweight than RabbitMQ.

Redis as a Celery broker distributed tasks extremely quickly and got the results back and even cached them nicely for us. Operations that would take 10 seconds with a RabbitMQ backend now took 1 second. RabbitMQ as a Celery broker distributed tasks a little bit slower and gets results back a little bit slower.

The obvious choice at this point would be to use Redis and get it over with, right?

Well, queuing up 500,000+ tasks on Redis degraded its performance immensely. RAM usage was still fine and the CPU wasn’t doing much work, but tasks just took much longer to get out and return - that same 1 second single task took 30 seconds when there were 500,000 other identical tasks sitting in the queue.

RabbitMQ was indifferent when we threw hundreds of thousands of tasks on it and performance was just the same as if we only gave it 10 tasks.

So we’re left with a problem - running 10 tasks on Redis is extremely fast, but running 500,000 tasks is extremely slow. Running 10 tasks with RabbitMQ yields okay performance, and running 500,000 tasks on it yields okay performance.

So right now, I’m still looking for other options or for a way to combine the two - we’ll figure it out.

Hello World!

2012-06-09T00:00:00+00:00

Hello - I’m Mike Staszel, a CS student at UIUC and a Linux developer, most notably known for being the founder of Arch Linux ARM.

Right now I’m working at SimpleRelevance - doing everything that falls into the broad category of Research and Development to build the next big thing.

This blog is the easiest-to-use blog I could have possibly imagined - using Jekyll Bootstrap and a neat auto-updating Git setup that generates HTML for posts from Markdown files when I run git push. I’ll briefly talk about it in a future post.