Working on a project where I need to grab screenshots of pages on a schedule — mostly for QA and keeping a visual history of what changed and when.
Right now I’m running Puppeteer in a cron job on a cheap VPS. It works, but honestly it’s a pain to maintain. Chromium eats RAM like crazy, sometimes it hangs and the whole thing needs a restart, and the screenshots come out wrong on pages that load content dynamically.
A few things I’ve been struggling with:
- Pages with lazy-loaded images — half the time the screenshot fires before everything renders
- Cookie consent banners blocking the actual content
- Memory usage goes through the roof when I try to do more than ~50 pages in a batch
I’ve looked into Playwright as a replacement but from what I can tell the resource usage is about the same. Also tried running headless Chrome in Docker which at least makes cleanup easier, but didn’t solve the core problems.
Curious what others are using. Are screenshot APIs worth it for this kind of thing, or is self-hosting still the way to go? Anyone running something similar at scale?
I haven’t used Puppeteer but have seen similar issues with Selenium and Firefox. Yeah you have to code delays for pages to load, use onload events, maybe scan the page for text you expect to see, etc. If this is for your own site (I mean where you’re working) you can probably do most QA functions by having some special flags that disable the overlays and stuff. Also if you have disk space to burn, instead of screen shots, you can connect the browser frame buffer to ffmpeg and capture video. That can be useful for QA, since if something goes wrong, you can see an actual movie of what was happening.
Yes browsers eat ram, you just have to deal with that.
An extra observation, if you’re doing QA and you’re running out of memory, there is a good chance that the actual users will experience the same issue.
If you’re running this inside Docker, you can specify exactly how much memory it has access to, which can also act as a data point for your QA process.
Pages with lazy-loaded images — half the time the screenshot fires before everything renders
Are you using waitUntil? You need to adjust what it’s waiting for. Networkidle2 works great for simple pages, but if there’s a lot of BS that never stops firing then it may never conder it loaded. I use domcontentloaded plus a hard coded delay.
Cookie consent banners blocking the actual content
Write a quick queryselector and have it click accept if it’s there.
Memory usage goes through the roof when I try to do more than ~50 pages in a batch
Not much you can do about that. Chrome just sucks. Any alternative is either going to use just as much ram, or not be representative of the real world. So you’re stuck with Chrome wearing different hats. Not using shitty hardware for it is amazing. We have some hosts that are almost 15 year old hardware and they’re painfully slow. But put on something modern with an ssd and it’s pretty fast. You just have to keep memory under control to keep it relatively cheap. You could probably buy an old laptop and run it as a server for about a years worth of decent VPS costs. As long as it’s got an ssd it’ll be fast enough for this.
Puppeteer is the tool. It takes some getting used to.
You can set a bunch of triggers that determine exactly when the screenshot fires, and you can simulate mouse movement and scrolling to deal with lazy loading.
There’s even time delays that differentiates between human time and machine time, allowing you to have the software act as if it was running for longer than it actually was, which deals with other weirdness seen in web apps.
You can run it inside Docker which might simplify things for you.
I ended up writing a node.js app to control puppeteer precisely how I needed to.
Source: I spent 48 months or so using it for a project that required all of that.
