🤑 Indie game store🙌 Free games😂 Fun games😨 Horror games
👷 Game development🎨 Assets📚 Comics
🎉 Sales🎁 Bundles

Butler fails to process specific archive

A topic by YellowAfterlife created 8 days ago Views: 43 Replies: 3
Viewing posts 1 to 4

So I'm trying to push a new channel with a Mac build, but it's failing to process with no further information.
The contents are nothing out of ordinary - an official pre-compiled Electron binary with app resources injected via 7-zip on Win10.

butler push Editor-Mac.zip yellowafterlife/gmedit:Editor-Mac
∙ For channel `editor-mac`: pushing first build
∙ Pushing 123 MiB (318 files, 157 dirs, 0 symlinks)
√ Added 123 MiB fresh data
√ 49 MiB patch (59.96% savings)
∙ Build is now processing, should be up in a bit (see `butler status`)


butler status yellowafterlife/gmedit:Editor-Mac
No channel editor-mac found for yellowafterlife/gmedit

Archive in question: https://www.dropbox.com/s/657isnnq5z42h2i/Editor-Mac.zip?dl=0
(also uploaded to project page as a non-butler build)

Admin

Ah, that's an interesting one! I took half an hour to investigate it.

lsar is a command that lets me list the contents of an archive.

$ lsar Editor-Mac.zip | head -5
Editor-Mac.zip: Zip
Electron.app/Contents/Resources/app/
Electron.app/Contents/Resources/app/ace/
Electron.app/Contents/Resources/app/ace/cleanup.bat
Electron.app/Contents/Resources/app/ace/mode-gml.js

When I count the number of entries in Editor-Mac.zip, I get:

$ lsar Editor-Mac.zip | wc -l
490

But if I sort the filenames and keep only the unique ones, I get:

$ lsar Editor-Mac.zip | sort -n | uniq | wc -l
409

This is consistent with other tests I've made - for example, diffing an empty folder against the zip (replicating what `butler push` does), then applying this patch (without verification, so it doesn't fail), then diffing Editor-Mac.zip against the newly-patched folder, and probing that patch:

$ butler probe zeropatch.pwr
∙ patch:  20 KiB
  before: 123 MiB in 318 files, 157 dirs, 0 symlinks
   after: 123 MiB in 248 files, 157 dirs, 0 symlinks
[...]

So it seems the `.zip` archive contains multiple entries with the same path name - I didn't realize this was even possible :)

I was able to get a complete list by doing:

$ lsar Editor-Mac.zip | sort -n | uniq -c -d
      2 Electron.app/Contents/Resources/app/
      2 Electron.app/Contents/Resources/app/ace/
      2 Electron.app/Contents/Resources/app/ace/cleanup.bat
      2 Electron.app/Contents/Resources/app/ace/mode-gml.js
[...]

Here's the full list if you're curious. The entries are also shown twice in 7-zip:


I'm happy that wharf caught that, but now there's two possible ways we could handle it:

  • Reject .zip files that contain multiple entries with the same name (I'm strongly tempted by that option)
    • This would've made the failure obvious when pushing the archive, and it would have had a proper error message, for example: Multiple entries with same filename found in .zip, bailing out (example: Electron.app/Contents/Resources/app/ace/cleanup.bat)
  • Or accept these files and make later entries overwrite previous ones - but I'm not in love with that option at all
    • First of all, I don't understand why 7-zip would ever generate a .zip archive with multiples entries of the same name
    • What if the version you actually want is the earlier entry and not the later one?
    • What if other archive extractors behave differently?

According to this StackOverflow topic, it's not a huge deal and we should be using the later ones. Hmm.

I assume that multiple entries might have been generating because of using the "rename" option - since 7-zip doesn't have an option to specify destination directory when adding files, it is commonly advised to add the files as-is and then move them wherever you want, so I did,

cd resources
7z a ../Editor-Mac.zip app
7z rn ../Editor-Mac.zip app Electron.app/Contents/Resources/app
cd ..

This is given directory structure like

Editor.exe (Windows binary)
Editor-Win.zip
Editor-Mac.zip
resources/
  app/ (has the actual changing files)

I'll see if there are better options - my concern about merging a Electron.app directory was that this overwrites file access permissions on it (so it's no longer +x) but having duplicate entries in the archive probably isn't much better.

Admin

So, permissions are a non-issue, because when you push with butler, it detects Linux & macOS binary files and fixes their permissions transparently :)

Now symlinks, that's another story..

But if your app bundle doesn't have any symlinks, you could just build it as a folder and pass that directly to butler.