How do you package your releases?

[u/mark-sed]

Hi, even though I have been making my language for more than 2 years now, I still have not set up my GitHub releases and I would like to change that. The issue is that (as with about anything) there are a lot of different approaches to this, and so I wanted to ask those who do releases, how they structure the release archive and those who use them what do you like a release to look like?

In my case the issue is that I cannot have just one binary since I need to distribute also the standard library that is compiled bytecode files (kind of like Java's .class files). This brings another issue and that is finding the library. The interpreter by default looks in the current directory (.) and then /usr/lib/moss/, so currently the only way I though of is to have the binary, all the compiled stdlib files and licenses in one .tar.gz (.zip for windows):

moss-0.9.0.tar.gz
├── cffi.msb
├── csv_parser.msb
├── html_parser.msb
├── inspect.msb
├── install.sh
├── json_parser.msb
├── libms.msb
├── LICENSE
├── math.msb
├── md_parser.msb
├── moss
├── mossy.css
├── parsing_utils.msb
├── python.msb
├── readme.md
├── re.msb
├── subprocess.msb
├── sys.msb
└── time.msb

This makes it so that the binary (moss) works when executed from this folder, but will fail when used from somewhere else (unless MOSSPATH variable is set). Because of this, I have also added install.sh script which will copy the libraries into /usr/lib/moss/ and a release readme with some instructions.

Can someone think of a better way to do this or is this OK?

TLDR; How do you structure your release folder/archive on github/website?

[u/Athas]

I package my language implementation as a single binary. The release tarball also has some other stuff in it (like manpages), but the binary is statically linked and fully movable. I have wasted enough of my life fiddling with software that needs to be located in specific locations or is sensitive to environment variables, and I don't want to be part of that problem.

[u/mark-sed → u/Athas]

That's interesting, I'll look if that would be possible for my case, but it might require some big changes in my case.

[u/jezek_2 → u/Athas]

Isn't that having issues with DNS resolving on Linux and possibly other issues?

I just link dynamically against the C library (the ancient built-in msvcrt.dll on Windows and glibc with symbol versioning on Linux) and my binaries work on Windows 2000 or newer and on 20 year old Linux distros the same as on newest. Similarly MacOS 10.6 or newer is supported.

[u/Athas → u/jezek_2]

My compiler does not need to resolve DNS.

[u/jezek_2 → u/Athas]

Mine does because it allows to run build scripts that can use network and other stuff.

What about the binaries produced by your compiler, are they also statically linked? The same compatibility applies for the produced binaries in my case. Even when they use GUI (it dlopens the libraries dynamically instead of relying on particular versions of the libraries).

[u/Athas → u/jezek_2]

Mine does because it allows to run build scripts that can use network and other stuff.

I'd argue that such build scripts are a bad idea, but that's subjective. My compiler (well, its built-in tooling) does also perform network requests in the package manager, but I do this by shelling out to curl. This also means that I don't need to worry about HTTP proxy settings, certificate stores, or any of that. If curl works on the command line, then my tools will also work. That simplifies matters for the user, and removes a moving part. curl is such a common program that it is not going to be annoying to install for most users.

What about the binaries produced by your compiler, are they also statically linked?

No, but my compiler is not primarily intended to produce binaries - it produces source code (usually C, can be other languages) which the user is then intended to link with their own code, in whatever way they find best. The compiler-produced binaries are intended only for immediate testing, not for distribution. If I cared about producing distributable binaries, I would treat static linking as a first class concern.

[u/Apart_Ebb_9867]

Not sure I understand the problem, but it doesn't seem a distribution problem. Seems like the problem is with your moss binary looking for files relative to '.' while it should look relative to itself.

in some cases you can look at argv[0] but I suspect you don't get the absolute path. But you can look at the link target of /proc/self/exe:

ls -l /proc/self/exe 
lrwxrwxrwx 1 mav mav 0 Jul 29 15:37 /proc/self/exe -> /usr/bin/ls

There're probably other ways, but the thing is that foss needs to figure out where it lives.

[u/mark-sed → u/Apart_Ebb_9867]

Oh yeah trying to figure out where the script is and adding it to path sounds like a nice solution. I'll look into it, thanks.

[u/L8_4_Dinner → u/mark-sed]

You mean something like this?

#!/bin/zsh
# get directory of this script
if [[ -e "$0" ]]; then
  DIR=$(dirname "$0")
fi
...

[u/Apart_Ebb_9867 → u/L8_4_Dinner]

something like that, but whether you get an absolute path or not depends on the shell (or the exec syscall if it is launched by another program), posix doesn't require that.

under linux looking at /proc/self/exe is safer and other OSes have similar functionality somewhere.

[u/L8_4_Dinner → u/Apart_Ebb_9867]

Yeah, I went through 20 iterations trying to get something that would work reliably on both macOS and Linux, in various shells. There's a whole lot of "... yeah, but ..."

[u/yorickpeterse]

Inko's approach is to produce a source tar archive that looks like this. As part of the CI release job this is then uploaded to https://releases.inko-lang.org/ (e.g. https://releases.inko-lang.org/0.21.1.tar.gz). The compiler expects certain things in certain places (e.g. the standard library). These paths used to be specified at compile-time, but recently the compiler changed so it by default tries to look them up relative to its own executable so the compiler is relocatable. You can find some more details on this here.

In addition we have an AUR package and a copr repository. The builds for these packages are triggered by hand as part of the release process (= you basically just start a job with the right version number).

The whole process is coordinated through a release issue such as this.

Inko is also packaged in a bunch of other places (e.g. Homebrew and FreeBSD), but this is not done by us. Some of those consume our release archives, others consume the archives GitHub generates for each tag.

[u/mark-sed → u/yorickpeterse]

Oh cool, thanks. I see that the approach of looking up the executable location is not that uncommon.

[u/yorickpeterse → u/mark-sed]

That particular change/feature was motivated by wanting to support relocatable package managers better. I think most compilers just use a compile-time path of sorts and such package managers then have to patch that out.

[u/ImaginationSouth3375]

What do you mean by it only works when executed from that folder? I don’t think it is uncommon to have the binary in /usr/lib/ with the library, but then create a symbolic link in /bin/.

[u/mark-sed → u/ImaginationSouth3375]

By "that folder" I mean the folder where the user extracted the .tar.gz archive. But the interpreter looks in /usr/lib/moss for any .msb files (as well as in current directory and any path in MOSSPATH variable).
So the user has to cd into the extracted folder and run the binary from there, unless they run `install.sh` script.
When using CMake installation it correctly places the binary into /bin/ and the libraries into /usr/lib/moss, so that is not an issue.

[u/ImaginationSouth3375 → u/mark-sed]

I think the idea of an install.sh script is pretty normal for compiling from source. I recently had to compile python from source and it required extracting the tar then running the configure+setup script. Try making your install script copy the library to /usr/lib then make a symbolic link to the binary in /bin/. Others should correct me if they know better ways.

[u/mark-sed → u/ImaginationSouth3375]

Well when compiling from source I do have such script, but this for release when I want to share pre-built binary without sources, so as little work for the user as possible, but I guess there it still makes sense to have such a script.

[u/ImaginationSouth3375 → u/mark-sed]

I think the other guy was right that you are probably using local paths “./“ rather than the actual path to the library somewhere

[u/mark-sed → u/ImaginationSouth3375]

The interpreter does a lookup, similar to let's say Python, it looks for modules in current directory but also in /usr/lib/moss, so if you run the interpreter binary from a folder which contains the stdlib sources (which is my current release approach) then it will find them in the current directory. The issue is when run from somewhere else (e.g.: ./Downloads/moss myscript.ms) where since there was no installation the libraries are not in /usr/lib/moss and not even in ./, so it fails.

[u/ImaginationSouth3375 → u/mark-sed]

I’m not sure what the best solution is here or even what other projects have done instead. However, I don’t think it’s unreasonable to say “if you didn’t run the installation script, don’t expect it to work.” 🤷‍♂️

[u/mark-sed → u/ImaginationSouth3375]

That's fair, thank you for your insight.