Pushing a Zip archive combined with --if-changed switch doesn't work on Linux

A topic by simulatoralive created Jun 17, 2020 Views: 668 Replies: 6

Viewing posts 1 to 3

My build script does the following:

Compiles my game
Packs everything required into a Zip archive
Publishes this archive to itch via the butler command

Because I prefer to save time and bandwidth, if possible, my script uses the --if-changed switch. This worked well under Windows 7, last year when I was last actively working on this project. This year, I'm running from Linux Mint and butler chokes and fails with the following output:

• Comparing against previous build...
checking for differences: lstat /windows/Users/simulatoralive/Documents/Projects/Programming/simulatoralive/BlockGameEngine/bigblockengine-0.21.7-Alpha-bin.zip/bigblockengine/mods: not a directory

I can assure you that the directory in question is definitely a directory. It doesn't fail when I remove the --if-changed switch, which is what I'll be doing for the time being. I even tried switching to an older version of butler matching the version I used under Windows and this did not work either.

I've had an irritating time testing and investigating this bug, because the --dry-run switch does not perform the file comparisons like one might expect it to when combined with --if-changed. What I ended up doing was making a pair of hidden test channels to test publishing to, to investigate further.

This bug isn't too big of a deal, but it is a bit frustrating. The main reason I was using the --if-changed switch is that most of the files in my game's archive are support libraries and their required libraries. These don't normally change from one upload to the next and my changes are often quite small by comparison. I would prefer not to have to wait for an upload of close to ten megs for a tiny, one-line code change for a bug fix.

This time and bandwidth savings is actually the reason I switched to using butler over manual uploads.

Thank you.

Edit: Here's a link to the game in question: https://simulatoralive.itch.io/bigblockengine

The file "bigblockengine-win-linux-osx-alpha.zip" is the one you'd want to look at if you want to analyze in detail what went wrong.

Like Reply

Afloof.devModerator4 years ago

It would really help if you provided the commands that you’ve inserted in your build script. From what I can see in a glance, this line that you posted seems questionable:

/bigblockengine/mods: not a directory

From what I can tell, you passed “/bigblockengine/mods” as a parameter, which is an Absolute Path, not a Relative Path. That means it’s not looking for it in the current directory (which would look something like “/home/<your_username>/<path_to_your_project/bigblockengine/mods”), but it’s instead looking for it at “/bigblockengine/mods”.

This would be fixed if you removed the first slash and it became “bigblockengine/mods”, so that it will look for that folder locally. But unless you post the command that your script is using, I can’t guarantee this is the solution, this is the only thing I saw that could have caused it.

Like Reply

simulatoralive4 years ago (2 edits)

The complete path, as seen in my first post (this is all on the same line, regardless of how the browser may word-wrap):

/windows/Users/simulatoralive/Documents/Projects/Programming/simulatoralive/BlockGameEngine/bigblockengine-0.21.7-Alpha-bin.zip/bigblockengine/mods

You'll notice the small bit you quoted is only the very end of it.

Edit: The butler command is assembled from variables in my build script (which you can find in the source archive, by the way), but should have looked like this, for the release in question:

butler push --userversion=0.21.7-Alpha --if-changed --noprogress bigblockengine-0.21.7-Alpha-bin.zip

Like Reply

Afloof.devModerator4 years ago

Ah I misunderstood the formatting, sorry for that.

I took a look at the butler command you provided. Have you taken a look at the documentation of how to use butler’s “push” command? It is here: https://itch.io/docs/butler/pushing.html

Based on that documentation and the command that you posted, you are missing the “user/game:channel” argument, which tells butler who you are, and which game of yours should the build be applied to.

In your case “user/game:channel” should be something like “simulatoralive/bigblockengine:linux”. I used “linux” as the “channel”, but feel free to pick your own.

Let me know if this was the issue, also if you are replying to me, please use the “Reply” button on my comment, so that I get a notification and I can reply to you. I accidentally found this response by checking up manually if there was any update :)

Like Reply

simulatoralive4 years ago (1 edit)

It's in the build file. I just failed to copy it to my post. That is NOT the problem.

The channel is "simulatoralive/bigblockengine:win-linux-osx-Alpha". That should have been on the end of the command string you requested.

As stated in my first post, my build script worked just fine under Windows 7, with the --if-changed switch still in place. Under Linux, it does NOT work, because butler chokes and dies on an empty directory. Please, could someone look at fixing this bug? Additionally, I would appreciate it if the --dry-run switch were modified to actually do the complete file comparisons. It does not currently do this.

I would have made a bug report through butler's github page, save for the fact that I flat out refuse to ever use github so long as Microsoft is the owner. I posted here as an alternative.

Dark Dimension, before you continue to nit-pick at what I'm saying here on this forum, look at my build script; I'm posting a copy in a code block at the end of this post (this is not the complete version and has everything stripped out of it except the variables and the publish targets; if you insist on seeing the complete version go download the source archive for my game and look for the file "bigblockengine/build.xml" in that archive). This Ant build script is what actually does the job. The publish targets are at the very end. The variables are all clearly laid out at the beginning (each is a "property" XML tag, which for some unknown reason the forum has expended into start and finish tags; not how it should be, but whatever, the forum software is a control freak about code blocks) and are all surrounded by curly braces, preceded by a dollar sign, in any text where they get substituted in. The build target you'd want to look at specifically is publish-test. It's using a different channel, but that's a hidden channel I setup to further test this bug.

In Ant scripts, the "exec" task calls another program, in this case, butler, then add's the "line" attributes from any nested "arg" tags on as the parameters, after it has parsed variables out of them.

As you can see, the ONLY differences between the "publish" and "publish-test" targets is the --if-changed switch and the channel. The one with --if-changed, which is the old way I was doing things, chokes and dies.

To make myself very plain, I posted here to get the developer's attention. I believe that would be Amos, yes? May I please have his attention on this?

<project name="bigblockengine" default="jar" basedir=".">
  <!-- set global properties for this build -->
  <property name="name" value="${ant.project.name}"></property>
  <property name="name-verbose" value="Big Block Engine"></property>
  <property name="ver-major" value="0"></property>
  <property name="ver-minor" value="23"></property>
  <property name="ver-bug" value="2"></property>
  <property name="ver-flag" value="Alpha"></property>
  <property name="version" value="${ver-major}.${ver-minor}.${ver-bug} ${ver-flag}"></property>
  <property name="version2" value="${ver-major}.${ver-minor}.${ver-bug}-${ver-flag}"></property>
  <property name="main-class" value="net.sf.simulatoralive.blockgame.GameLoader"></property>
  <property name="doc-packages" value="*"></property>
  <property name="doc-title" value="${name-verbose} ${version}"></property>
  <property name="source-version" value="1.8"></property>
  <property name="target-version" value="1.8"></property>
  <!-- Files/directories for this build -->
  <property name="doc-root" location="doc"></property>
  <property name="doc" location="${doc-root}/${name}"></property>
  <property name="src-root" location="src"></property>
  <property name="src" location="${src-root}/${name}"></property>
  <property name="src-bootstrap" location="${src-root}/bootstrap"></property>
  <property name="bootstrap-zip" location="mods-zip/bootstrap.game.zip"></property>
  <property name="temp-dir" location="classes"></property>
  <property name="build" location="${temp-dir}/${name}"></property>
  <property name="dist" location="."></property>
  <property name="dist-jar" location="${dist}/${name}.jar"></property>
  <property name="dist-tar" location="${name}-${version2}.tar.bz2"></property>
  <property name="dist-tar-bin" location="${name}-${version2}-bin.tar.bz2"></property>
  <property name="dist-zip-bin" location="${name}-${version2}-bin.zip"></property>
  <property name="dist-tar-src" location="${name}-${version2}-src.tar.bz2"></property>
  <property name="dist-zip-src" location="${name}-${version2}-src.zip"></property>
  <!-- Classpath for compilation -->
  <property name="classpath" value="lib/hierarchy.jar;lib/jogl/jogl-all.jar;lib/jogl/gluegen-rt.jar;lib/substance.jar;lib/trident.jar;lib/svg-salamander.jar;lib/dyn4j.jar"></property>
  <!-- Jar file classpath is space separated -->
  <property name="jar-classpath" value="lib/hierarchy.jar lib/jogl/jogl-all.jar lib/jogl/gluegen-rt.jar lib/substance.jar lib/trident.jar lib/dyn4j.jar"></property>
  <property name="butler-bin-channel" value="simulatoralive/bigblockengine:win-linux-osx-${ver-flag}"></property>
  <property name="butler-src-channel" value="simulatoralive/bigblockengine:source"></property>
  <property name="butler-bin-channel-test" value="simulatoralive/bigblockengine:test-${ver-flag}"></property>
  <property name="butler-src-channel-test" value="simulatoralive/bigblockengine:test-source"></property>
  <property name="butler-switches" value="--userversion=${version2} --noprogress"></property>
  
  <target name="publish" depends="dist-bin,dist-src">
    <!-- Push the binary build using itch.io Butler command.
         Hopefully this is cross-platform enough to work. -->
    <echo message="Publishing binary build..." level="info"></echo>
    <exec executable="butler" failonerror="true">
      <arg line="push ${butler-switches} ${dist-zip-bin} ${butler-bin-channel}"></arg>
    </exec>
    <echo message="Binary build published." level="info"></echo>
    
    <!-- Push the sources using itch.io Butler command.
         Hopefully this is cross-platform enough to work. -->
    <echo message="Publishing sources..." level="info"></echo>
    <exec executable="butler" failonerror="true">
      <arg line="push ${butler-switches} ${dist-zip-src} ${butler-src-channel}"></arg>
    </exec>
    <echo message="Sources published." level="info"></echo>
  </target>
  
  <target name="publish-test" depends="dist-bin,dist-src">
    <!-- Push the binary build using itch.io Butler command.
         Hopefully this is cross-platform enough to work. -->
    <echo message="Publishing binary build..." level="info"></echo>
    <exec executable="butler" failonerror="true">
      <arg line="push --if-changed ${butler-switches} ${dist-zip-bin} ${butler-bin-channel-test}"></arg>
    </exec>
    <echo message="Binary build published." level="info"></echo>
    
    <!-- Push the sources using itch.io Butler command.
         Hopefully this is cross-platform enough to work. -->
    <echo message="Publishing sources..." level="info"></echo>
    <exec executable="butler" failonerror="true">
      <arg line="push --if-changed ${butler-switches} ${dist-zip-src} ${butler-src-channel-test}"></arg>
    </exec>
    <echo message="Sources published." level="info"></echo>
  </target>
  
</project>

Like Reply

Afloof.devModerator4 years ago

Sorry if I sounded like I was nit-picking. I was trying to find what the issue was with the limited information that was available.

I work on Linux, and use butler without an issue, including the “–if-changed” flag, so I assumed something else might be causing this issue. I’m not sure what is causing this, maybe someone else can help.

Like Reply

simulatoralive4 years ago

My tests have convinced me that this is a bug in Butler and a very subtle one, at that. That's why I wanted the developer that wrote it to see my posts.

Like Reply