Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
Tags

Butler fails to process specific archive

A topic by YellowAfterlife created Jan 11, 2018 Views: 1,171 Replies: 8
Viewing posts 1 to 5

So I'm trying to push a new channel with a Mac build, but it's failing to process with no further information.
The contents are nothing out of ordinary - an official pre-compiled Electron binary with app resources injected via 7-zip on Win10.

butler push Editor-Mac.zip yellowafterlife/gmedit:Editor-Mac
∙ For channel `editor-mac`: pushing first build
∙ Pushing 123 MiB (318 files, 157 dirs, 0 symlinks)
√ Added 123 MiB fresh data
√ 49 MiB patch (59.96% savings)
∙ Build is now processing, should be up in a bit (see `butler status`)


butler status yellowafterlife/gmedit:Editor-Mac
No channel editor-mac found for yellowafterlife/gmedit

Archive in question: https://www.dropbox.com/s/657isnnq5z42h2i/Editor-Mac.zip?dl=0
(also uploaded to project page as a non-butler build)

Ah, that's an interesting one! I took half an hour to investigate it.

lsar is a command that lets me list the contents of an archive.

$ lsar Editor-Mac.zip | head -5
Editor-Mac.zip: Zip
Electron.app/Contents/Resources/app/
Electron.app/Contents/Resources/app/ace/
Electron.app/Contents/Resources/app/ace/cleanup.bat
Electron.app/Contents/Resources/app/ace/mode-gml.js

When I count the number of entries in Editor-Mac.zip, I get:

$ lsar Editor-Mac.zip | wc -l
490

But if I sort the filenames and keep only the unique ones, I get:

$ lsar Editor-Mac.zip | sort -n | uniq | wc -l
409

This is consistent with other tests I've made - for example, diffing an empty folder against the zip (replicating what `butler push` does), then applying this patch (without verification, so it doesn't fail), then diffing Editor-Mac.zip against the newly-patched folder, and probing that patch:

$ butler probe zeropatch.pwr
∙ patch:  20 KiB
  before: 123 MiB in 318 files, 157 dirs, 0 symlinks
   after: 123 MiB in 248 files, 157 dirs, 0 symlinks
[...]

So it seems the `.zip` archive contains multiple entries with the same path name - I didn't realize this was even possible :)

I was able to get a complete list by doing:

$ lsar Editor-Mac.zip | sort -n | uniq -c -d
      2 Electron.app/Contents/Resources/app/
      2 Electron.app/Contents/Resources/app/ace/
      2 Electron.app/Contents/Resources/app/ace/cleanup.bat
      2 Electron.app/Contents/Resources/app/ace/mode-gml.js
[...]

Here's the full list if you're curious. The entries are also shown twice in 7-zip:


I'm happy that wharf caught that, but now there's two possible ways we could handle it:

  • Reject .zip files that contain multiple entries with the same name (I'm strongly tempted by that option)
    • This would've made the failure obvious when pushing the archive, and it would have had a proper error message, for example: Multiple entries with same filename found in .zip, bailing out (example: Electron.app/Contents/Resources/app/ace/cleanup.bat)
  • Or accept these files and make later entries overwrite previous ones - but I'm not in love with that option at all
    • First of all, I don't understand why 7-zip would ever generate a .zip archive with multiples entries of the same name
    • What if the version you actually want is the earlier entry and not the later one?
    • What if other archive extractors behave differently?

According to this StackOverflow topic, it's not a huge deal and we should be using the later ones. Hmm.

Trying to understand, are the files' data actually in the ZIP multiple times? Or just the path listing? If the files are actually duplicated, I would say to reject these because they are potentially adding a lot of duplicated data, but if not, no problem.

Then again, your system takes care of such an issue with how it handles compression anyhow, does it not?

(+1)

I think the data is actually duplicated (in the OP's zip) - instead of making a new zip, it just appends the new files to the end. When extracting, whichever entry occurs last should be used.

It wouldn't matter once it's uploaded with butler though, because a fresh .zip is made on the backend  (that never has duplicates).

I assume that multiple entries might have been generating because of using the "rename" option - since 7-zip doesn't have an option to specify destination directory when adding files, it is commonly advised to add the files as-is and then move them wherever you want, so I did,

cd resources
7z a ../Editor-Mac.zip app
7z rn ../Editor-Mac.zip app Electron.app/Contents/Resources/app
cd ..

This is given directory structure like

Editor.exe (Windows binary)
Editor-Win.zip
Editor-Mac.zip
resources/
  app/ (has the actual changing files)

I'll see if there are better options - my concern about merging a Electron.app directory was that this overwrites file access permissions on it (so it's no longer +x) but having duplicate entries in the archive probably isn't much better.

So, permissions are a non-issue, because when you push with butler, it detects Linux & macOS binary files and fixes their permissions transparently :)

Now symlinks, that's another story..

But if your app bundle doesn't have any symlinks, you could just build it as a folder and pass that directly to butler.

Hi, I get the same error processing message, mine is not a new push but my 21st build update. The previous all worked fine and the patch difference is 81MB only. Status says its processing the new build but then the error occurs and it falls back to version 20. I wanted to run Verify but cannot seem to find documentation on how the path should look for the signature file.

butler push D:\Upload ahavahstudio/the-prometheus-secret-noohra:windows-beta

 

Hey there, there were some server issues earlier, but they've been fixed and all failed builds have now been processed correctly.

(+1)

Thank you Amos for the quick response, all seems in order.