pacman/scripts/libmakepkg/source/git.sh.in
Eli Schwartz 35a0d5e744 makepkg: use "shared" git clones when checking out sources
In order to cache sources offline, makepkg creates *two* copies of every
git repo. This is a useful tradeoff for network time, but comes at the
cost of increased disk space.

Normally, git can smooth this over automagically. Whenever possible, git
objects are hardlinked to save space, but this does not work when
SRCDEST and BUILDDIR are on separate filesystems.

When the repo in question is both very large (linux.git for example is
2.2 GB) and crosses filesystem boundaries, this results in a lot of
extra disk space being used; the most likely scenario is where BUILDDIR
is a tmpfs for bonus ouch.

git(1) has a builtin feature which serves this case handily: the
--shared flag will create the info/alternates file instructing git to
not copy or hardlink or create objects/packs at all, but merely look for
them in an external location (that being the source of the clone).

The downside of using shared clones, is that if you modify and drop
commits from the original repo, or simply delete the whole repo
altogether, you break the copy. But we don't care about that here,
because

1) the BUILDDIR copy is meant to be a temporary copy strictly derived
   via PKGBUILD syntax from the SRCDEST, and must be able to be
   recreated at any time,
2) if the SRCDEST disappears, makepkg will redownload it, thus restoring
   the objects needed by the BUILDDIR clone,
3) if the user does non-default things like hacking on the BUILDDIR copy
   then deleting and re-cloning the SRCDEST may result in momentary
   breakage, but ultimately should be fine -- the unique objects they
   created will be stored in the BUILDDIR copy.

While it's theoretically possible that upstream will force-push to
overwrite the base tree from which makepkg is building (which they
should not do), *and* the user deleted their SRCDEST which they should
not do, *and* they saved work in makepkg's working directory which they
should not do either...
... this is an unlikely chain of events for which we should not care.

Using --shared is therefore helpful in immediately useful ways and IMHO
has no actual downsides; we should use it.

An alternative implementation would be to use worktrees. I've rejected
this since it is essentially the same as shared clones, except adding
additional restrictions on the branch namespace, and could potentially
break existing use cases such as manually handling the SRCDEST in order
to share repositories with normal working copies.

Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
2019-03-19 14:09:00 +10:00

136 lines
3.8 KiB
Bash

#!/bin/bash
#
# git.sh - function for handling the download and "extraction" of Git sources
#
# Copyright (c) 2015-2018 Pacman Development Team <pacman-dev@archlinux.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
[[ -n "$LIBMAKEPKG_SOURCE_GIT_SH" ]] && return
LIBMAKEPKG_SOURCE_GIT_SH=1
LIBRARY=${LIBRARY:-'@libmakepkgdir@'}
source "$LIBRARY/util/message.sh"
source "$LIBRARY/util/pkgbuild.sh"
download_git() {
# abort early if parent says not to fetch
if declare -p get_vcs > /dev/null 2>&1; then
(( get_vcs )) || return
fi
local netfile=$1
local dir=$(get_filepath "$netfile")
[[ -z "$dir" ]] && dir="$SRCDEST/$(get_filename "$netfile")"
local repo=$(get_filename "$netfile")
local url=$(get_url "$netfile")
url=${url#git+}
url=${url%%#*}
url=${url%%\?*}
if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then
msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git"
if ! git clone --mirror "$url" "$dir"; then
error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git"
plain "$(gettext "Aborting...")"
exit 1
fi
elif (( ! HOLDVER )); then
cd_safe "$dir"
# Make sure we are fetching the right repo
if [[ "$url" != "$(git config --get remote.origin.url)" ]] ; then
error "$(gettext "%s is not a clone of %s")" "$dir" "$url"
plain "$(gettext "Aborting...")"
exit 1
fi
msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git"
if ! git fetch --all -p; then
# only warn on failure to allow offline builds
warning "$(gettext "Failure while updating %s %s repo")" "${repo}" "git"
fi
fi
}
extract_git() {
local netfile=$1 tagname
local fragment=$(get_uri_fragment "$netfile")
local repo=$(get_filename "$netfile")
local dir=$(get_filepath "$netfile")
[[ -z "$dir" ]] && dir="$SRCDEST/$(get_filename "$netfile")"
msg2 "$(gettext "Creating working copy of %s %s repo...")" "${repo}" "git"
pushd "$srcdir" &>/dev/null
local updating=0
if [[ -d "${dir##*/}" ]]; then
updating=1
cd_safe "${dir##*/}"
if ! git fetch; then
error "$(gettext "Failure while updating working copy of %s %s repo")" "${repo}" "git"
plain "$(gettext "Aborting...")"
exit 1
fi
cd_safe "$srcdir"
elif ! git clone -s "$dir" "${dir##*/}"; then
error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git"
plain "$(gettext "Aborting...")"
exit 1
fi
cd_safe "${dir##*/}"
local ref=origin/HEAD
if [[ -n $fragment ]]; then
case ${fragment%%=*} in
commit|tag)
ref=${fragment##*=}
;;
branch)
ref=origin/${fragment##*=}
;;
*)
error "$(gettext "Unrecognized reference: %s")" "${fragment}"
plain "$(gettext "Aborting...")"
exit 1
esac
fi
if [[ ${fragment%%=*} = tag ]]; then
tagname="$(git tag -l --format='%(tag)' "$ref")"
if [[ -n $tagname && $tagname != $ref ]]; then
error "$(gettext "Failure while checking out version %s, the git tag has been forged")" "$ref"
plain "$(gettext "Aborting...")"
exit 1
fi
fi
if [[ $ref != "origin/HEAD" ]] || (( updating )) ; then
if ! git checkout --force --no-track -B makepkg $ref; then
error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git"
plain "$(gettext "Aborting...")"
exit 1
fi
fi
popd &>/dev/null
}