Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
Tags

Jav-It

The most comprehensive JAV video cataloging tool. · By jav-it

Pls add gui interface

A topic by stivify created May 01, 2021 Views: 867 Replies: 18
Viewing posts 1 to 4

It will be good to have a gui interface instead of command prompt so that we can select all the features with a few clicks? 

Developer (1 edit) (+1)

Thanks for the suggestion! Perhaps a front end could be created for the tool, but I haven't put much thought into it since other features seem to be more important to people (Ie. Machine Language Translation of subtitle files, etc). In the meantime, if you need help with the command line, let me know and I'd be happy to assist you. Because most of this is done automatically, the commands you need are usually extremely simple! (^_^);

Edit: Patreon requests are always prioritized at the top of my backlog, so if your really serious about getting a GUI/frontend solution, perhaps consider subscribing to my Patreon. I like to try and help everyone, but since my time is rather limited, I need to focus my effort where it will benifit the project the most. 

(1 edit)

Thanks for your reply! Right now I still got many issues getting this to work for me. There are many files with names 'hhd800.com@' at the front and the program cannot search from the actual DVD ID. I have to manually edit the names but it will be too time consuming. There are few hundreds of these. I hope I can retain the original file names so that I can continue to seed.

And can these work for Plex?  I tried loading with Plex but many metadata was not displayed correctly.

Developer (1 edit)

Hi,

Ah, your getting this becuse "hhd800" looks like a video ID, so Jav-It is trying to parse it first rather than continue to look within the filename for additional patterns. Since your file name format is rather interesting, It was never really considered. I think I have an idea on how to deal with this edge case, so I'll see if I can get a fix prepared for you.

As for original filenames, there is an option to leave them as originally found (and not change them), but that only effects the filename and not the foldernames. Is that enough for you? Preserving torrent names with Symbolic Links is pretty easy on Linux/MacOS, but I'm not sure if Windows has something similar. Perhaps If I know more about what your torrent requirements are, I can try to find a solution. 

As for Plex, I'm not sure because I don't have it (Since Plex is not free). The NFO file format follows KODI specifications, which is also compatable with other open source projects like JellyFin (A FOSS Plex alternative). I assume if you have an importer or program that can read KODI formatted files then you would be fine. 

Developer

Just a small update: I think I have a working solution for your issue. So instead of detecting domains that have names similar to video IDs, it will just ignore it. I should be able to get a release out maybe tonight, but I want to make sure it's well tested first (^_^); 

Developer

Ok, I created a silent hotfix for your issue. If you re-download version 1.4.1 again, give it a try and see if scanning picks up the right ID for you now. Let me know how it goes. I appreciate you bringing this to my attention! If people don't let me know, I can't investigate. (^_^);

(2 edits)

Hi the fix works for me now.  Thank u so much! It does picks up the right ID for those files. As for the leaving the original filenames I have tried using the code --alt-naming-method="original" it does effect on both filenames and foldernames so there is no problem now.

As for Plex there is a free version if u no need the extra features. I did use an importer called XBMCnfoMoviesImporter to load the KODI formatted nfo and it does load most of the items except for the actor image. Since it doesn't load can just have the option to disable downloading of actor image? Another reason is because there will be too many duplicate actor images under different folders as each actor can have many DVDs. To serve different users requirement u can fix to allow display of actors image for Plex and also disable downloading altogether? There is no need to have a tag genre for 'JAV' since the entire folder is JAV, hope I can disable it. Also same goes to the .tbn file which is essentially the folder.jpg in different format only use for KODI. In essence I only require the poster, fanart, thriller, nfo and everything else can remain as original. As for the title it will be good to add the DVD ID on the front so the I can search via DVD ID. Something like MIDE-123 Title...

Lastly if can rollback on the items that was done that will be good. ^^

Developer (1 edit)

Thanks for the feedback, as I’m glad it works for you now!

In regards to actor images, adding an option to ignore them is totally possible. It just needs to be prioritized and added to the backlog. The reason why images are mirrored for each folder is due to technical limitations. Actor images must be found locally or be specified by a fully qualified path. Since there is no way to guarantee the location where people choose to store such information, locally is the best option. In addition, since actor images are limited to head shots, the disk space requirement for them is quite low.

As for the “JAV” genre tag, this is important for other edge-cases. For example, many users have collections that include JAV, but are not exclusively all JAV. So when adding a small amount of JAV content to their libraries, having accurate genre tags is desirable for searching.

In regards to thumbnail files, indeed some of the metadata included is exclusive to KODI standards. While this information might take up a marginal amount of disk space, since KODI has been the most dominant and popular open source media center project for a while it makes sense to align compatibility with it. This is especially true, since other projects tend to use subsets from the KODI specification, meaning general compatibility can be maintained.

I hope this clarifies things a bit! (^_^);

(4 edits)

Thanks for your clarifications. Not sure why is Plex isn't displaying the actor image when it is stored locally.  But if can scrape the nfo to include the thumb url instead it can display on both KODI and Plex so downloading the image will become optional. For example like this below.

<actor>               
  <name>Nishimiya Yume</name>                     
  <thumb>https://pics.r18.com/mono/actjpgs/nisimiya_yume.jpg</thumb>    
  <role>Actress</role>       
</actor>

The interface for Plex and KODI is very different. Plex is more suitable for PC using with keyboard and mice whereas KODI is more for TV with remote. They are both very dominant and have their own userbase. Plex is a great enhancement for KODI for those who prefer PC. It has more metadata to display, better sorting and filtering etc. You should explore Plex and give it a try, if possible support for both. Thanks!

Developer

Ah, thanks for kind update! Hrm... I could make actor images have a toggle to use the URL instead of local copy. However, the reason why I provide a local copy is so that places like R18 cannot secretly take down content or decommision titles and cause you to loose your entries (However unlikely that is).

Let me do some testing and I'll see what I can come up with. (^_^);

Developer (2 edits)

Hi Stivify,

I just wanted to let you know that I reviewed your request. Now, in addition to making a local mirror, the <thumb> tag is also populated. So if you import it into applications, such as Plex which do not respect all aspects of the KODI standard, actress photos should still show up like normal. All you need to do is download the new 1.5.1 version to pickup the change. (^_^);

(3 edits)

Hello jav-it I use Plex as well, they have some good documentation about setting up local metadata if you want to have a look. 

https://support.plex.tv/articles/200220677-local-media-assets-movies/

https://support.plex.tv/articles/local-files-for-trailers-and-extras/

if Kodi and Plex have different file structure maybe an option can be made to pick what one to use Kodi or Plex?

Developer (1 edit)

Hi,

Thanks for the documentation! This is something that looks quite feasible. Perhaps something to look at after machine translation! 

Thanks jav-it for the actor image. A step closer to making it Plex friendly. I am almost certain that Plex is a better option for JAV. Just look at the screen capture below the amount of metadata it can display, specify, sort, filter and categorise. The below is using a Plex agent called PlexJav18.bundle GitHub - alxpnt2/PlexJav18.bundle scraping method which does not add additional files to our local drive. I have compared both, however it has limitations which includes:

1. it doesn't keep a copy in your drive just in case the metadata got erased in r18 (unlikely but possible). 

2. It does not include trailer. 

3. The folder image is lesser resolution than jav-it which will look blur when expanded. 

4. It doesn't change the censored title ***** into actual words.

So jav-it have complemented on these which is great but still need tweaks to make it perfect. i.e. The title to include DVD-ID for easier sorting and searching. There is really no need to have a tag called "JAV" because it will look like a main category instead of sub-category in Plex. We can add a main category ourselves by adding another movie tab, which will look cleaner if we want to have multiple types of other movies in Plex. ^^

Developer (1 edit)

Hi,

Thanks for the feedback. It sounds like Plex could totally be added as an additional output format for scraping. Given some helpfull feedback I got from another thread, it sounds like as long as there is solid documentation as to what plex expects, it could be implemented with a little thought. I just need to plan a bit. (҂⌣̀_⌣́). So how exactly does KODI imported metadata look in Plex anyway? Do you have any clear screenshots? This isn't a small task, so if any work is done on this front, it will have to be after I get machine translation out the door (which is still in progress). It was delayed due to all that Patreon nonsense. (>_<);

In regards to a tag called "JAV", this is important to some users because they do not exclusively collect JAV. So in a very large collection of other videos, the "JAV" tag is important to them. I don't personally need the tag, but some people have indicated it was important so I included it. I don't know if it really hurts anything to be honest.

In regards to metadata disapearing, I can confirm that it does happen and suddenly! Because I need a large random subset of files to test with, It was easy to notice when R18 discontinued videos and pulled the metadata for it. It really sucks to be honest! It might not be widespread, but it looks awful when you have a full collection, but 1 or 2 videos are empty because they can no longer be scraped. 

(1 edit)

This is how KODI formatted looks like in Plex. Initially after running jav-it for the first time it seems ok with some minor issues like no DVD-ID and actor image. Subsequently running it over the same folder a few times (due to the new updates) many issues came. Titles become missing from the nfo. Some items got additional as the program thought that the trailer were also the actual movie and generate additional nfo fanart etc. The first time I didn't change to preserve original file names and subsequent runs it generates again and preserves files names but includes the previous generated items. Also when partially downloaded files ending with .bc will still be considered as a movie file and the items gets downloaded. After movie file download completed and running a second time the items gets re-downloaded. My local folder are also full of messy duplicate files. That's why I recommended that the jav-it can have a roll-back function to delete whatever generated previously before running it again so that such issues can be avoided.

current jav-it v1.51 in Plex. 


an incomplete file with .bc! file extension gets downloaded with the files

 extra files regenerated
extra files re-generated due to incomplete files with .bc! extension previously were detected as movie files


titles become missing in random nfo files 

As for the tag called "JAV" it is important to some users because they didn't sort their files correctly in the first place. Only JAV movies require jav-it to be scraped hence it doesn't make sense for them to include their other collection in the same folder. The default shouldn't include this tag. It looks weird having to organise in movie type yet another tag for the same movie type. It isn't important just that OCD people will get annoyed by it.

Developer

Hi,

Thanks for the nice screenshots! Oh boy, where to start! (^_^); I don't know what "*.bc!" files are, and where your getting them. If your saying that these files are moved into folders and being cataloged, then that's a bug because Jav-It should not look for any file with a non-movie extensions. When you say "an incomplete file with .bc! file extension gets downloaded with the files", I'm a little confused as to what you mean.

From your screenshot, I can see you have multiple files called "*.mp4.nfo", "*.mp4.tbn", etc. Where did those files come from initialy? Some torrent you downloaded? Or was that extra data added by Plex during the import? Or was that created by Jav-It? 

Perhaps to be clear, what was the name of the movie you tried to catalog before you ran Jav-It? "KCDA-296.mp4"? Or was it "KCDA-296.mp4.bc!" What were the exact comands you tried to run with Jav-It to get this undesired behaviour? Was it one command or many?

As to using alternate naming conventions with Jav-It, it is already warned in the command line help that existing cataloged files under different naming conventions are not removed. This is primarally a safety issue since deleting files is potentially destructive, especially if Jav-It has no idea how you tried to name things previously. Perhaps an override setting which blindly deletes everything except for the main video could be added as a cleaning mechanisim, but it would be a risky operation especially if your video is seperated into parts. For now, all I can recommend is do not run Jav-It multiple times using different naming conventions, unless you plan to manually review and clean files out yourself. Just pick one and stick to it. (^_^); For now anyways!

As for missing titles, if you can give me steps or something I can reproduce, I can definately try to understand where they are going. I'm pretty sure the multiple *.NFO file is causing your issue. If you simply choose a cataloging format and stick to it, then you'd likely be fine? Until I can program a cleaning switch, resetting yourelf will have to be done manually. But the easiest way I found is to simply do this:

- In your OS, do a find under your root folder for all *.jpg, *.tbn, *-trailer.*, .actors, and extrafanart, files and directories.
- Once you get a list of all the files found, just review them to make sure nothing was picked up by accident and delete them all in one shot.

Then you can start your scraping using a clean base. If your still getting issues after doing this, then let me know what steps your doing and I can try to reproduce what your seeing.

Yes ".bc!" files are bitcomet incomplete downloaded files. The bitcomet placed an extension to the files until it is completed. Technically it should not be detected by Jav-it as a movie file. Apparently Jav-it still detects it as movie file which is a bug. The multiple files are created by Jav-it using a single command. Those ".bc!" files get downloaded with "*.mp4.nfo" or "*.mp4.tbn". Subsequently after the files completed download by bitcomet and changed to .mp4 extension, running Jav-it the second time on these files (using the same command) will download additional files with "*.nfo" or "*.tbn". Plex do not add extra files to the import. Ideally stopping Jav-it from detecting ".bc!" extensions or any other extension other than movie extension will solve this issue. 

I only use this command, jav-it.exe catalog -d u:\Downloads --alt-naming-method="original" but it will come out multiple files from the same set of movies. For example "KCDA-296.mp4.bc!" was the initial file, running Jav-it will get "KCDA-296.mp4.nfo". Subsequently "KCDA-296.mp4", when running Jav-it again will have "KCDA-296.nfo". 

Yes under alternate naming conventions with Jav-It, existing cataloged files under different naming conventions are not removed is correct, but I'm using the same naming convention. Extra files still get generated but with a "*.mp4.nfo" naming. 

As for missing titles, I have no idea how to reproduce it because it happens randomly. I will try to delete all the files using your method and try to generate with a clean base and see how it goes.

Developer (2 edits)

Hi,

Thanks for the detailed description. Because I was able to clearly understand you, I have succesfully resolved the edge-case where *.bc! files are incorrectly being picked up as video files. Since the prefix is a valid video (and it was not foreseen users would try to catalog folders in a changing state), video files were being selected incorrectly based on a loose regex which has now been updated. You should be able to see change in the next version of Jav-It (Sometime this week or next). I suspect that with this change, all the extra files that are generated shold now go away. (^_^);

As for your missing titles, that one I am still unsure about. However, because of the strange edge-case you were trying to do, maybe that was the cause of it. There is a likely chance that by igoring *bc! files, a lot of this strangeness will go away.