So, what I want is to download lemmy posts (including comments) as raw html through a script. The problem I have is, that curl by default does not include the css of the page. Is there a way I can download the posts webpages as they are being displayed?
why html? just saving the json it fetches in the first place would be a lot easier to work with IMO, but it depends on what you’re doing with it
I second that - can’t you just use the API and request the post and its comments as json?
Unless you want to explicitly archive the post as a human viewable HTML, the structured json from the API is almost always superior to further analyze, archive,… the posts.
Archiving thwmin a human readable form is exactly what I want to do.
Then I’m not sure if curl is up to the task in this case - most modern websites are PWAs that dynamically request data from their Backen and display it. Curl just fetches the initial HTML but doesn’t execute the JS and thus might not see the content of the page as it isn’t there yet the moment it gets saved.
Some PWAs “render” the first page contents in your HTML so that it is included and do not need to be requested.
Maybe try the alternative frontends if some of them work better? Also remember that feddit.org and many other instances deploy Anubis and might lock you out from making simple requests with curl
Got it working with wget