🤑 Indie game store🙌 Free games😂 Fun games😨 Horror games
👷 Game development🎨 Assets📚 Comics
🎉 Sales🎁 Bundles

butler upload - symlinks problem!

A topic by notpatchman created 15 days ago Views: 80 Replies: 11
Viewing posts 1 to 9

Hello, I'm trying to upload builds of my game, but instead of butler following symlinks it uploads the actual symlinks themselves!

I'd really like to share my resources between different builds and not have many duplicates filling up my hard drive, not to mention the maintenance headache copying changes to every build.

Is there a way to tell butler not to "hard upload" symlinks? 

thanks!

Well, I've checked all documentation and looked at the source code.

Nope.  Doesn't look possible as is, since this behavior is hard-coded into butler.  :(  Which is really, really, REALLLLLLLLLLLLY annoying, but isn't a dealbreaker.  I thought there might be workarounds, like using butler to upload twice to the same repo but in different directories, which woudn't work. I can't think of a way.  I suppose I can temporarily move game data to each build before uploading every time, which is less of a headache, but still frustrating.

Admin (Edited 1 time)

Heyo,

Talking about symlinks is always confusing so let's take something sane as a reference, like the GNU tar documentation:

Normally, when tar archives a symbolic link, it writes a block to the archive naming the target of the link. In that way, the tar archive is a faithful record of the file system contents. When `--dereference' (`-h') is used with `--create' (`-c'), tar archives the files symbolic links point to, instead of the links themselves.

That's also butler's behavior. It tries to be a faithful record of the file system contents.

Having that behavior by default is really useful because "dereferencing" symlinks makes (for example) macOS builds unnecessarily large - those often have symlinks in `YourApp.app/Contents/MacOS/Frameworks`. Linux builds also tend to have symlinks, although it's less standard (obviously!)

Since butler/wharf are entirely itch.io codebases, we can definitely add a `--dereference` option for your usecase - it won't be the default, but it still might make things better for you?

Hi!!

Yes, that option would be incredibly handy.  I agree that it should not be the default - otherwise you'll break a lot of people's builds!

For Linux, which is where I do all my development, and will upload every platform build from to itch.io, the symlinks are super handy. 

Adding that option would definitely work for me, thanks for considering my plea.  :)

Admin

the bleeding-edge version of butler (butler head) now accepts a `--dereference` option to the `push` command:

butler push --dereference somedirectory user/game:channel

You can check what it sees ahead of time by using the (hidden) walk command:

butler --json walk --dereference somedirectory

To get the latest bleeding-edge butler, you can run

butler upgrade --head

Let me know how it works for you!

Such service!  Wow.  Trying...

(Edited 2 times)

Hmm. I get an error with both commands. :(

  • cd ~/Games/patchman/release/itch.io
  • ./butler --json walk --dereference linux
  • os.PathError lstat /home/dev/Games/patchman/release/itch.io/linux/home/dev/Games/patchman/release/q1-assets-release4comics/art: no such file or directory

Looks like the path string connector logic has failed.  It's appending the symlink to the directory path.  These should be separate:

  • /home/dev/Games/patchman/release/itch.io/linux
  • /home/dev/Games/patchman/release/q1-assets-release4comics/art

The first one is the directory to upload, the second is the followed symlink.

Edit: It shouldn't even try to resolve any symlinks at all.  There is no reason (in this case) to know about "q1-assets-release4comics/art", it should just be "itch.io/linux/art" where "art" was the symlinked dir.  Hope this didn't create a confusing mess for you!!

Admin

Ah, I think I see what the issue is. I've been testing with relative symlinks, not absolute :)

Admin

I just released a new (bleeding-edge) build of butler that should handle absolute symlinks properly.

(Edited 1 time) (+1)

It works as expected, I've tested with my Linux build! 

Will also need to try osx+windows (uploaded from Linux), but I don't forsee any problems.

Thank you Amos!!  You made my life a bit easier :)

Admin

Glad it's working for you, enjoy!

(+1)

Update: been uploading shared resources to diff builds with no problems. 

Smooth!!