Post by jhice in Big Improvements to Library-To-CSV Scraper comments

Big Improvements to Library-To-CSV Scraper » Comments

Viewing post in Big Improvements to Library-To-CSV Scraper comments

Hi abraxas86 and thanks a lot for this script ! I was looking such a solution and you've done it.

Just tested it, it works like a charm.

The major need is to have a nice way to manage collections as it is not yet possible on Itch. With your script, any Webdev could do its own work upon it and do what they want. As I want to do also.

I looked how the data is pulled from Itch and yes, it's all HTML from the request and the API does not permit this kind of access (like JSON results as we'd like to have). BUT there are fetch requests like these ones : https://itch.io/my-collections/more-games/9211?page=2 that bring the same HTML that you parse in your Tampermonkey script, so your script could probably be updated to automatically get all the collections in one click ! Yes, it's more work to do. I'm not asking you to code it, it's definitely something I or someone else can try too.

Ideally, an Electron app (or even a Website) could to the job :p (again I'm not asking you to make it ^^). So the app could be shared to every Itch user.

Your script is a very good start and job !
I'll keep you informed if I work on it.

Edit : For a standalone app, I realise the credentials thing could be tricky to manage as the collection URL is not part of the API. Perhaps using the OAuth login we could make requests with the logged user, not sure, to be continued...

abraxas86294 days ago

Hey, thanks for the kind words! Glad this script was useful for you! I was surprised when I first went on the hunt to figure out how to export the collections - I thought for sure there would be an option somewhere on the site and I was just missing it; but in my hunt, all I could find were a few posts of other people asking how to do it (and being told it couldn't be done).

Nice catch on that API fetch! I'm still pretty new to all this stuff so I wouldn't have thought to go digging for something like that... but maybe I'll see if I can find a way to take advantage of it. I think the biggest hurdle will be figuring out how many pages there are, but I guess I could find that by iterating through the pages until I get some sort of error back. 🤔

I've got a few things on the go so I can't tackle another update to this yet, but I'll keep it in mind :)

jhice293 days ago(+1)

Hey :) Glad to have a reply from you too ! (and played and commented my last game jam entry ^^ will reply to it later ;))

I'm on my way to grab all the informations at once, if you let me let's say until the end of september, I could probably show you the result (I have other things to do too, and not much free time). For the moment I managed to grab the information from the fetch call and use your parser to save the data, but not in a loop for now.

At the end, perhaps it will be another script, perhaps the script could have 2 purposes : "Export all" on the main page and "Export collection" on the page collection ¯\_(ツ)_/¯ (if we can export all at once, it would be probably useless to export just a category).

What do you think ?

Cheers !

PS : the correct URL is more like this one that the one I put in my first message : https://itch.io/c/{collection_id}/{collection_slug}?page=2&format=json

abraxas86292 days ago

Sounds good to me! If I get some extra time to mess around with this in the meantime, I'll let you know :)