Been dealing with this more often lately. Tests pass on my machine, I push, and CI blows up. Usually it’s one of these:
- Different Node/Python/whatever version
- Missing env vars that exist in my .env but not in CI secrets
- File system case sensitivity (macOS vs Linux)
- Some flaky test that depends on timing
My current debugging flow is pretty basic: check the logs, compare versions, run the exact same Docker image locally if I can. But it still eats 20-30 minutes each time before I figure out the actual problem.
Anyone have a more systematic approach? Like a quick checklist you run through before you even look at the logs?
Also curious — do you replicate your CI environment locally with something like act (for GitHub Actions) or just trust the remote runner?
If you have a known good (local) and known bad (remote), the first thing I’d probably do is run diff on the local and remote logs. Use a regex or something to strip timestamps or similar that will always differ.
If your tests take 20–30 minutes to run, I think that I’d look into having a way to run a subset of the tests, so you can just run the minimal amount to do the failing one.
If you have a lot of environment problems, I think I’d focus on getting an automated build of the environment to a standard state. If that takes too long, setting up caching or whatever of downloaded packages.