are you using mongo db or something like that on the backend.
how often do you hit the sites?
i wonder how you parse the sites.
i had written a quick cron/php program to text me (max 3 times/day) when i could find 9mm ammo available on some websites.
we were lucky enough to get a database guy to help us that worked on a failed search engine in the early 2000's. He brought a ton of knowledge and we are actually running a custom "gunwatcher" version of mysql with integrate sphinx (which is how we parse 100,000's records when you search so quickly and also keep our out of pocket cost down). We also are running a lot of servers that search throughout the day. For black Friday the OLDEST data can possibly be is one hour. Most deals should be caught within 20min on average. The parsing is another program entirely written by a few other of our programmers that is several 1000's of lines of code which has data cleaning, and a ton of other useful things before it finally hits the db, which is another server in itself.