diff options
Diffstat (limited to 'contrib/gitian')
| -rw-r--r-- | contrib/gitian/README.md | 125 | ||||
| -rwxr-xr-x | contrib/gitian/gitian-build.py | 35 | ||||
| -rw-r--r-- | contrib/gitian/gitian-linux.yml | 13 | ||||
| -rw-r--r-- | contrib/gitian/gitian-osx.yml | 12 | ||||
| -rw-r--r-- | contrib/gitian/gitian-win.yml | 12 |
5 files changed, 133 insertions, 64 deletions
diff --git a/contrib/gitian/README.md b/contrib/gitian/README.md index 1efc87e0a..32aee5f56 100644 --- a/contrib/gitian/README.md +++ b/contrib/gitian/README.md @@ -26,12 +26,16 @@ Preparing the Gitian builder host The first step is to prepare the host environment that will be used to perform the Gitian builds. This guide explains how to set up the environment, and how to start the builds. -Gitian offers to build with either `kvm`, `docker` or `lxc`. The default build -path chosen is `lxc`, but its setup is more complicated. You need to be logged in as the `gitianuser`. -If this user does not exist yet on your system, create it. Gitian can use -either kvm, lxc or docker as a host environment. This documentation will show -how to build with lxc and docker. While the docker setup is easy, the lxc setup -is more involved. + +* Gitian host OS should be Ubuntu 18.04 "Bionic Beaver". If you are on a mac or windows for example, you can run it in a VM but will be slower. + +* Gitian gives you the option of using any of 3 different virtualization tools: `kvm`, `docker` or `lxc`. This documentation will only show how to build with `lxc` and `docker` (documentation for `kvm` is welcome). Building with `lxc` is the default, but is more complicated, so we recommend docker your first time. + + +## Create the gitianuser account + +You need to create a new user called `gitianuser` and be logged in as that user. The user needs `sudo` access. + LXC --- @@ -76,18 +80,34 @@ This setup is required to enable networking in the container. Docker ------ -Building in docker does not require much setup. Install docker on your host, then type the following: +Prepare for building with docker: + +```bash +sudo apt-get install git make curl docker.io +``` + +Consider adding `gitianuser` to the `docker` group after reading about [the security implications](https://docs.docker.com/v17.09/engine/installation/linux/linux-postinstall/): ```bash -sudo apt-get install git make curl +sudo groupadd docker sudo usermod -aG docker gitianuser ``` +Optionally add yourself to the docker group. Note that this will give docker root access to your system. -Manual and Building +```bash +sudo usermod -aG docker gitianuser +``` + +Manual Building ------------------- The instructions below use the automated script [gitian-build.py](gitian-build.py) which only works in Ubuntu. +======= +The script automatically installs some packages with apt. If you are not running it on a debian-like system, pass `--no-apt` along with the other +arguments to it. It calls all available .yml descriptors, which in turn pass the build configurations for different platforms to gitian. +The instructions below use the automated script [gitian-build.py](gitian-build.py) which is tested to work on Ubuntu. + It calls all available .yml descriptors, which in turn pass the build configurations for different platforms to gitian. Help for the build steps taken can be accessed with `./gitian-build.py --help`. @@ -100,66 +120,95 @@ The `gitian-build.py` script will checkout different release tags, so it's best cp monero/contrib/gitian/gitian-build.py . ``` -Setup the required environment, you only need to do this once: +### Setup the required environment + +Setup for LXC: ```bash -./gitian-build.py --setup fluffypony v0.14.0 +GH_USER=fluffypony +VERSION=v0.14.1.0 + +./gitian-build.py --setup $GH_USER $VERSION ``` -Where `fluffypony` is your Github name and `v0.14.0` is the version tag you want to build. -If you are using docker, run it with: +Where `GH_USER` is your Github user name and `VERSION` is the version tag you want to build. + +Setup for docker: ```bash -./gitian-build.py --setup --docker fluffypony v0.14.0 +./gitian-build.py --setup --docker $GH_USER $VERSION ``` -While gitian and this build script does provide a way for you to sign the build directly, it is recommended to sign in a seperate step. -This script is only there for convenience. Seperate steps for building can still be taken. +While gitian and this build script does provide a way for you to sign the build directly, it is recommended to sign in a separate step. This script is only there for convenience. Separate steps for building can still be taken. In order to sign gitian builds on your host machine, which has your PGP key, -fork the gitian.sigs repository and clone it on your host machine, +fork the [gitian.sigs repository](https://github.com/monero-project/gitian.sigs) and clone it on your host machine, or pass the signed assert file back to your build machine. ```bash git clone git@github.com:monero-project/gitian.sigs.git -git remote add fluffypony git@github.com:fluffypony/gitian.sigs.git +git remote add $GH_USER git@github.com:$GH_USER/gitian.sigs.git ``` -Build Binaries ------------------------------ -To build the most recent tag (pass in `--docker` after setting up with docker): +Build the binaries +------------------ + +**Note:** if you intend to build MacOS binaries, please follow [these instructions](https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-mac-os-sdk.md) to get the required SDK. + +To build the most recent tag (pass in `--docker` if using docker): ```bash -./gitian-build.py --detach-sign --no-commit -b fluffypony v0.14.0 +./gitian-build.py --detach-sign --no-commit --build $GH_USER $VERSION ``` -To speed up the build, use `-j 5 -m 5000` as the first arguments, where `5` is the number of CPU's you allocated to the VM plus one, and 5000 is a little bit less than then the MB's of RAM you allocated. If there is memory corruption on your machine, try to tweak these values. +To speed up the build, use `-j 5 --memory 5000` as the first arguments, where `5` is the number of CPU's you allocated to the VM plus one, and 5000 is a little bit less than then the MB's of RAM you allocated. If there is memory corruption on your machine, try to tweak these values. + +If all went well, this produces a number of (uncommitted) `.assert` files in the gitian.sigs directory. + +Checking your work +------------------ + +Take a look in the assert files and note the SHA256 checksums listed there. eg for `v0.14.1.0` you should get this checksum: -If all went well, this produces a number of (uncommited) `.assert` files in the gitian.sigs repository. +``` +2b95118f53d98d542a85f8732b84ba13b3cd20517ccb40332b0edd0ddf4f8c62 monero-x86_64-linux-gnu.tar.gz +``` -If you do detached, offline signing, you need to copy these uncommited changes to your host machine, where you can sign them. For example: +You should verify that this is really the checksum you get on that file you built. You can also look in the gitian.sigs repo and / or [getmonero.org release checksums](https://web.getmonero.org/downloads/hashes.txt) to see if others got the same checksum for the same version tag. If there is ever a mismatch -- **STOP! Something is wrong**. Contact others on IRC / github to figure out what is going on. + + +Signing assert files +-------------------- + +If you chose to do detached signing using `--detach-sign` above (recommended), you need to copy these uncommitted changes to your host machine, then sign them using your gpg key like so: ```bash -export NAME=fluffypony -export VERSION=v0.14.0 -gpg --output $VERSION-linux/$NAME/monero-linux-$VERSION-build.assert.sig --detach-sign $VERSION-linux/$NAME/monero-linux-$VERSION-build.assert -gpg --output $VERSION-osx-unsigned/$NAME/monero-osx-$VERSION-build.assert.sig --detach-sign $VERSION-osx-unsigned/$NAME/monero-osx-$VERSION-build.assert -gpg --output $VERSION-win-unsigned/$NAME/monero-win-$VERSION-build.assert.sig --detach-sign $VERSION-win-unsigned/$NAME/monero-win-$VERSION-build.assert +GH_USER=fluffypony +VERSION=v0.14.1.0 + +gpg --detach-sign ${VERSION}-linux/${GH_USER}/monero-linux-*-build.assert +gpg --detach-sign ${VERSION}-win/${GH_USER}/monero-win-*-build.assert +gpg --detach-sign ${VERSION}-osx/${GH_USER}/monero-osx-*-build.assert ``` +<!-- TODO: Replace * above with ${VERSION} once gitian builds correct file name --> + +This will create a `.sig` file for each `.assert` file above (2 files for each platform). + + +Submitting your signed assert files +----------------------------------- Make a pull request (both the `.assert` and `.assert.sig` files) to the [monero-project/gitian.sigs](https://github.com/monero-project/gitian.sigs/) repository: ```bash -git checkout -b v0.14.0 -git commit -S -a -m "Add $NAME v0.14.0" -git push --set-upstream $NAME v0.14.0 +git checkout -b $VERSION +# add your assert and sig files... +git commit -S -a -m "Add $GH_USER $VERSION" +git push --set-upstream $GH_USER $VERSION ``` -```bash -gpg --detach-sign ${VERSION}-linux/${SIGNER}/monero-linux-*-build.assert -gpg --detach-sign ${VERSION}-win-unsigned/${SIGNER}/monero-win-*-build.assert -gpg --detach-sign ${VERSION}-osx-unsigned/${SIGNER}/monero-osx-*-build.assert -``` +**Note:** Please ensure your gpg public key is available to check signatures by adding it to the [gitian.sigs/gitian-pubkeys/](https://github.com/monero-project/gitian.sigs/tree/master/gitian-pubkeys) directory in a pull request. + More Build Options ------------------ diff --git a/contrib/gitian/gitian-build.py b/contrib/gitian/gitian-build.py index cd88ecb20..b654b15c7 100755 --- a/contrib/gitian/gitian-build.py +++ b/contrib/gitian/gitian-build.py @@ -7,21 +7,13 @@ import sys def setup(): global args, workdir - programs = ['ruby', 'git', 'apt-cacher-ng', 'make', 'wget'] + programs = ['apt-cacher-ng', 'ruby', 'git', 'make', 'wget'] if args.kvm: programs += ['python-vm-builder', 'qemu-kvm', 'qemu-utils'] - elif args.docker: - dockers = ['docker.io', 'docker-ce'] - for i in dockers: - return_code = subprocess.call(['sudo', 'apt-get', 'install', '-qq', i]) - if return_code == 0: - break - if return_code != 0: - print('Cannot find any way to install docker', file=sys.stderr) - exit(1) else: programs += ['lxc', 'debootstrap'] - subprocess.check_call(['sudo', 'apt-get', 'install', '-qq'] + programs) + if not args.no_apt: + subprocess.check_call(['sudo', 'apt-get', 'install', '-qq'] + programs) if not os.path.isdir('gitian.sigs'): subprocess.check_call(['git', 'clone', 'https://github.com/monero-project/gitian.sigs.git']) if not os.path.isdir('gitian-builder'): @@ -32,6 +24,8 @@ def setup(): subprocess.check_call(['git', 'checkout', '963322de8420c50502c4cc33d4d7c0d84437b576']) make_image_prog = ['bin/make-base-vm', '--suite', 'bionic', '--arch', 'amd64'] if args.docker: + if not subprocess.call(['docker', '--help'], shell=False, stdout=subprocess.DEVNULL): + print("Please install docker first manually") make_image_prog += ['--docker'] elif not args.kvm: make_image_prog += ['--lxc'] @@ -40,7 +34,7 @@ def setup(): if args.is_bionic and not args.kvm and not args.docker: subprocess.check_call(['sudo', 'sed', '-i', 's/lxcbr0/br0/', '/etc/default/lxc-net']) print('Reboot is required') - exit(0) + sys.exit(0) def build(): global args, workdir @@ -100,7 +94,7 @@ def verify(): def main(): global args, workdir - parser = argparse.ArgumentParser(usage='%(prog)s [options] signer version') + parser = argparse.ArgumentParser(description='Script for running full Gitian builds.', usage='%(prog)s [options] signer version') parser.add_argument('-c', '--commit', action='store_true', dest='commit', help='Indicate that the version argument is for a commit or branch') parser.add_argument('-p', '--pull', action='store_true', dest='pull', help='Indicate that the version argument is the number of a github repository pull request') parser.add_argument('-u', '--url', dest='url', default='https://github.com/monero-project/monero', help='Specify the URL of the repository. Default is %(default)s') @@ -112,11 +106,12 @@ def main(): parser.add_argument('-m', '--memory', dest='memory', default='2000', help='Memory to allocate in MiB. Default %(default)s') parser.add_argument('-k', '--kvm', action='store_true', dest='kvm', help='Use KVM instead of LXC') parser.add_argument('-d', '--docker', action='store_true', dest='docker', help='Use Docker instead of LXC') - parser.add_argument('-S', '--setup', action='store_true', dest='setup', help='Set up the Gitian building environment. Uses LXC. If you want to use KVM, use the --kvm option. Only works on Debian-based systems (Ubuntu, Debian)') + parser.add_argument('-S', '--setup', action='store_true', dest='setup', help='Set up the Gitian building environment. Uses LXC. If you want to use KVM, use the --kvm option. If you run this script on a non-debian based system, pass the --no-apt flag') parser.add_argument('-D', '--detach-sign', action='store_true', dest='detach_sign', help='Create the assert file for detached signing. Will not commit anything.') parser.add_argument('-n', '--no-commit', action='store_false', dest='commit_files', help='Do not commit anything to git') - parser.add_argument('signer', help='GPG signer to sign each build assert file') - parser.add_argument('version', help='Version number, commit, or branch to build.') + parser.add_argument('signer', nargs='?', help='GPG signer to sign each build assert file') + parser.add_argument('version', nargs='?', help='Version number, commit, or branch to build.') + parser.add_argument('-a', '--no-apt', action='store_true', dest='no_apt', help='Indicate that apt is not installed on the system') args = parser.parse_args() workdir = os.getcwd() @@ -128,8 +123,8 @@ def main(): args.is_bionic = b'bionic' in subprocess.check_output(['lsb_release', '-cs']) if args.buildsign: - args.build=True - args.sign=True + args.build = True + args.sign = True if args.kvm and args.docker: raise Exception('Error: cannot have both kvm and docker') @@ -156,11 +151,11 @@ def main(): if args.signer == '': print(script_name+': Missing signer.') print('Try '+script_name+' --help for more information') - exit(1) + sys.exit(1) if args.version == '': print(script_name+': Missing version.') print('Try '+script_name+' --help for more information') - exit(1) + sys.exit(1) # Add leading 'v' for tags if args.commit and args.pull: diff --git a/contrib/gitian/gitian-linux.yml b/contrib/gitian/gitian-linux.yml index 67f174fec..fd94d43bf 100644 --- a/contrib/gitian/gitian-linux.yml +++ b/contrib/gitian/gitian-linux.yml @@ -129,7 +129,15 @@ script: | chmod +x ${WRAP_DIR}/${prog} done + git config --global core.abbrev 9 cd monero + # Set the version string that gets added to the tar archive name + version="`git describe`" + if [[ $version == *"-"*"-"* ]]; then + version="`git rev-parse --short=9 HEAD`" + version="`echo $version | head -c 9`" + fi + BASEPREFIX=`pwd`/contrib/depends # Build dependencies for each host for i in $HOSTS; do @@ -153,10 +161,11 @@ script: | export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH} mkdir build && cd build cmake .. -DCMAKE_TOOLCHAIN_FILE=${BASEPREFIX}/${i}/share/toolchain.cmake -DBACKCOMPAT=ON - make - DISTNAME=monero-${i} + make ${MAKEOPTS} + DISTNAME=monero-${i}-${version} mv bin ${DISTNAME} find ${DISTNAME}/ | sort | tar --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}.tar.gz cd .. rm -rf build done + diff --git a/contrib/gitian/gitian-osx.yml b/contrib/gitian/gitian-osx.yml index 7de302353..77ea30072 100644 --- a/contrib/gitian/gitian-osx.yml +++ b/contrib/gitian/gitian-osx.yml @@ -77,7 +77,15 @@ script: | create_per-host_faketime_wrappers "2000-01-01 12:00:00" export PATH=${WRAP_DIR}:${PATH} + git config --global core.abbrev 9 cd monero + # Set the version string that gets added to the tar archive name + version="`git describe`" + if [[ $version == *"-"*"-"* ]]; then + version="`git rev-parse --short=9 HEAD`" + version="`echo $version | head -c 9`" + fi + BASEPREFIX=`pwd`/contrib/depends mkdir -p ${BASEPREFIX}/SDKs @@ -100,8 +108,8 @@ script: | export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH} mkdir build && cd build cmake .. -DCMAKE_TOOLCHAIN_FILE=${BASEPREFIX}/${i}/share/toolchain.cmake - make - DISTNAME=monero-${i} + make ${MAKEOPTS} + DISTNAME=monero-${i}-${version} mv bin ${DISTNAME} find ${DISTNAME}/ | sort | tar --no-recursion --mode='u+rw,go+r-w,a+X' --owner=0 --group=0 -c -T - | gzip -9n > ${OUTDIR}/${DISTNAME}.tar.gz cd .. diff --git a/contrib/gitian/gitian-win.yml b/contrib/gitian/gitian-win.yml index 1eb558300..4c559acfe 100644 --- a/contrib/gitian/gitian-win.yml +++ b/contrib/gitian/gitian-win.yml @@ -100,7 +100,15 @@ script: | create_per-host_linker_wrapper "2000-01-01 12:00:00" export PATH=${WRAP_DIR}:${PATH} + git config --global core.abbrev 9 cd monero + # Set the version string that gets added to the tar archive name + version="`git describe`" + if [[ $version == *"-"*"-"* ]]; then + version="`git rev-parse --short=9 HEAD`" + version="`echo $version | head -c 9`" + fi + BASEPREFIX=`pwd`/contrib/depends # Build dependencies for each host for i in $HOSTS; do @@ -125,8 +133,8 @@ script: | export PATH=${BASEPREFIX}/${i}/native/bin:${ORIGPATH} mkdir build && cd build cmake .. -DCMAKE_TOOLCHAIN_FILE=${BASEPREFIX}/${i}/share/toolchain.cmake - make - DISTNAME=monero-${i} + make ${MAKEOPTS} + DISTNAME=monero-${i}-${version} mv bin ${DISTNAME} find ${DISTNAME}/ | sort | zip -X@ ${OUTDIR}/${DISTNAME}.zip cd .. && rm -rf build |