Self Hosted Stock Checking

In the times of COVID-19, many things sold online are currently out of stock. Some people enjoy constantly hitting F5 on webpages, but for the rest of us there has to be a better way.

There are a couple of browser extension options and cloud-based options such as Distill.io which work great, but cost money or require a browser to be left open and running. Anyone who runs a home server has a couple more efficient options:

• Write a script to periodically wget a web page, parse the html and look at the appropriate tags. This is great and quick, but if you have dozens of things to check can get cumbersome.
• Run Huginn and create an “agent” to monitor specific pages.

We’ll take a quick look at Huginn here.

I run a NAS in my home setup where most of my home services run as docker containers. Luckily the Huginn team has two deployable containers already up on the docker hub. The multi-process container (the standard install) and the single-process (used for scalability of individual agents). See here for more details. So, all we have to do is pull huginn/huggin from the docker hub, set up a volume to /var/lib/mysql so we can back up the database easily, then choose which port to map to the container’s web UI (by default 3000).

Once that is up and running we can create new “Website Agent” that will monitor a web page for changes. I won’t go into detail on the configuration, but I’ll post my example configuration for monitoring for a REP weight tree.

{
  "expected_update_period_in_days": "2",
  "url": "https://www.repfitness.com/bars-plates/storage/plates/bar-and-bumper-plate-tree",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "stock": {
      "xpath": "//*[@id=\"maincontent\"]/div[2]/div/div[2]/div[1]/div[1]/div[3]/div[1]",
      "value": "."
    }
  }
}

I got the xpath by simply opening up the developer tools in Chrome, using the pointer tool to highlight the “Out of Stock” element, then right clicking the element and going to Copy -> XPath.

As another example, here’s a configuration I made to check the status of the Walmart page for a PS5. Happy gaming!

{
  "expected_update_period_in_days": "2",
  "url": "https://www.walmart.com/ip/PlayStation5-Console/363472942",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "stock": {
      "xpath": "/html/body/div",
      "value": "."
    }
  },
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"
}