Commit graph

7127 commits

Author SHA1 Message Date
morganamilo
cde562bb4d
Merge branch 'morganamilo/pactrans' into morganamilo/tip 2021-11-07 00:19:35 +00:00
morganamilo
15eb65c839
Merge branch 'morganamilo/backup' into morganamilo/tip 2021-11-06 23:46:38 +00:00
morganamilo
40f59fd6ac
Merge branch 'morganamilo/email' into morganamilo/tip 2021-11-06 23:41:36 +00:00
morganamilo
3c68d11f3c
Merge branch 'morganamilo/qbackup' into morganamilo/tip 2021-11-06 23:41:29 +00:00
morganamilo
2200dd8797
Merge branch 'morganamilo/notes' into morganamilo/tip 2021-11-06 23:41:15 +00:00
morganamilo
c7c280115f
Merge branch 'morganamilo/noserver' into morganamilo/tip 2021-11-06 23:41:05 +00:00
morganamilo
564fd18c9a
Merge branch 'morganamilo/nolockdownload' into morganamilo/tip 2021-11-06 23:40:57 +00:00
morganamilo
5c55ef2db5
Merge branch 'morganamilo/nokeep' into morganamilo/tip 2021-11-06 23:40:45 +00:00
morganamilo
de5fb9ae2f
Merge branch 'morganamilo/nodownload' into morganamilo/tip 2021-11-06 23:40:34 +00:00
morganamilo
d4b4d315aa
Merge branch 'morganamilo/lint' into morganamilo/tip 2021-11-06 23:39:30 +00:00
morganamilo
97d127588e
Merge branch 'morganamilo/file_error' into morganamilo/tip 2021-11-06 23:39:17 +00:00
morganamilo
c301327ad5
pacman: implement universal transactions
This allows using -S, -U and -R in one command:

    pacman -S foo -R bar

To make this work some breaking changes have made.

Targets have to come after the operation:

    pacman -S foo  //works
    pacman -Syu    //works
    pacman -yuS    //works
    pacman foo -S  //doesn't work

This could be supported with some code to copy all targets before the
first operation into the first operation's target list.

And -u as a short for --unneeded has been removed as it conflicts with
--sysupgrade. However has -u/--sysupgrade is bound to -S, accidently
doing `pacman -Ru` will not accidently cause a system upgrade.

Another quirk with the ui is that -S has many non transaction related
flags, -Sc -Sg -Sl -Si. These have been split off as "sync only" flags.
Meaning they show up with `pacman -Si foo` but will be invalid on
`pacman -R bar -Si foo`.

Also when -R'ing and -S'ing the same package in come command it's
treated as a full uninstall then reinstall. The backup files are
.pacsave'd and the install reason is set to explicit. I feel like this
behavious is good. This also allows you to wipe config files which what
--nokeep was intending to solve.

Other flags just have to have the op they belong to to be used for them
to be valid.

For example:

    pacman -Rn foo //works
    pacman -S -Rn //works
    pacman -Sn    //doesn't work
    pacman -Sn -R //works

We could posibly drop these flags belonging to each operation and just
make them generic transaction flags.

Implements FS#9694
2021-11-06 23:29:07 +00:00
morganamilo
841236a1c9
pacman: improve backup printing
The current backup printing does not fit in with the rest of the info at
all. Change to be more consistant.

Old:

Backup Files    :
MODIFIED	/etc/pacman.conf
UNMODIFIED	/etc/makepkg.conf

New:

Backup Files    : /etc/pacman.conf [modified]
                  /etc/makepkg.conf [unmodified]

Signed-off-by: morganamilo <morganamilo@archlinux.org>
2021-10-23 22:50:39 +01:00
morganamilo
6ce85053c5
libalpm: allow nolock transactions when downloadonly 2021-10-23 22:19:46 +01:00
morganamilo
e877472509
makepkg: lint empty arrays
While depend arrays are already linted, many array kinds are
still not. An empty string is never a valid array value so check
all arrays for it.
2021-10-07 22:26:50 +01:00
morganamilo
2b4c022925
libalpm: add iterator interface for syncdb files
This commit adds an iterator interface for reading files from the
syncdbs. Instead of using alpm_pkg_get_files(), you now get the files
from the database using alpm_db_files_open(), you then use
alpm_db_files_next() to iterate through the files for each package. If
you want to actually load the files from that package you then use
alpm_db_files_load().

This means alpm_pkg_get_files() will always return empty for syncdbs,
even on .files databases, however these functions still work on the
localdb and loaded packages.

This aproach is faster when dumping the entire file list but slower when
searching for a specific package.

The memory usage of pacman is drastically less. See below.

build/pacman -Fl        0.55s user 0.01s system 99% cpu 0.556 total
build/pacman -Fl pacman 0.46s user 0.01s system 99% cpu 0.472 total
build/pacman -Fx pacman 2.88s user 0.09s system 99% cpu 2.965 total

pacman -Fl              1.60s user 0.13s system 99% cpu 1.731 total
pacman -Fl pacman       0.24s user 0.04s system 99% cpu 0.283 total
pacman -Fx pacman       2.45s user 0.14s system 99% cpu 2.593 total

                         Peak Memory
build/pacman -Fl         43.52MB
build/pacman -Fl pacmam  11.292MB

pacman -Fl               677.048MB
pacman -Fl pacman        163.288MB
2021-10-06 08:18:55 +01:00
morganamilo
689223f40d
alpm: don't download files from local servers
This causes file:// servers to be treated as if they were cache dirs
when checking if a package needs to be downloaded/read.
2021-10-04 20:11:24 +01:00
morganamilo
8d48332069
alpm: return -1 for error in find_dl_candidates
This is the error value generally used and the calling function
explicitly checks for -1, later causing the error to be missed
and the transaction to continue.

> pacman -S xterm
warning: xterm-369-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)  Old Version  New Version  Net Change  Download Size

extra/xterm  369-1        369-1          0.00 MiB       0.42 MiB

Total Download Size:   0.42 MiB
Total Installed Size:  1.05 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n]
error: no servers configured for repository: extra
(1/1) checking keys in keyring                                                                 [--------------------------------------------------------] 100%
(1/1) checking package integrity                                                               [--------------------------------------------------------] 100%
error: failed to commit transaction (wrong or NULL argument passed)
Errors occurred, no packages were upgraded.
2021-10-04 20:07:06 +01:00
morganamilo
a61c500557
pacman: add -Q --backup
pacman -Q -w/--backup will print the modified backup files of a system
(passing twice will print all backup files). This could be useful for
backup/moving system config files.
2021-10-02 00:12:52 +01:00
morganamilo
8fa9a69683
pactest: fix test errors being treated as success
Rules return -1 if there was an error with the rule itself. Later this
return value is passed to tap as a bool. Because -1 is a truthy value it
gets treated as success.
2021-09-27 21:46:13 +01:00
morganamilo
1340b5336e
pactest: add note tests 2021-09-27 21:46:10 +01:00
morganamilo
2b1ca6c298
pacman: add --note --rmnote
This adds --note for -S -U -D and --rmnote for -D
2021-09-27 21:46:07 +01:00
morganamilo
ef05b4e31e
alpm: add note support
Add support for adding a note to packages. This is intended to be set to
the user to document the reason or motive a package was installed.

Notes can be set for a transaction and only the targets of that
transaction gain the note.

Notes can also be edited for installed packages similarly to how install
reason can be set.
2021-09-27 21:46:04 +01:00
morganamilo
c7c88f880a
libalpm: check filecache_find return and log errors
Some user had erros while updating their system.

:: Proceed with installation? [Y/n]
:: Retrieving packages...
checking keyring...
checking package integrity...
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.

The issue was filecache_find returning null and alpm passing that null
path to check validity. How this happened I have no idea. It may be
something to do with the user's cachedir being a network drive.

Also warn when the file exists but it is not a regular file or can not
be opened.
2021-09-27 18:30:17 +01:00
morganamilo
753083d241
Update mailing list url
change pacman-dev@archlinux.org to pacmandev@lists.archlinux.org

Most of this is copyright notices but this also fixes FS#72129 by
updating the address in docs/index.asciidoc.

---

I could also update the email in the .po files as it's a simple find and
replace but I'm not sure if that's strictly done via transifex.
2021-09-22 14:51:11 +01:00
morganamilo
36d3d8401c
pacman: move --nosave to transaction flags
--nosave is now useful when installing packages as it can be combined
with --nokeep to reinstall the packages backup files without generating
a pacsave.
2021-09-20 20:34:43 +01:00
morganamilo
ddcceb9314
pacman: add --nokeep 2021-09-20 20:34:40 +01:00
morganamilo
0c701af590
alpm: add ALPM_TRANS_FLAG_NOKEEP
this flag prevents backup files from being kept on package installation.
This is useful for resetting a package's config files back to their
original state.

Implements FS#59908 although with it's own flag name instead of reusing
nosave. This allows nokeep to optionally create a pacnew that you can
then choose to disable by also setting nosave.

---

I actually very dislike NOKEEP but it was the best I could come up with

I would have prefered overwrite or nosave but they are taken. Better
names are welcome.
2021-09-20 20:31:33 +01:00
Allan McRae
39c3cbdf56 _alpm_key_import: Initialise fetch_key
Prevents build warning.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-05 09:58:18 +10:00
morganamilo
165e492485 pacman: don't run hooks when using --dbonly
--dbonly is meant to only touch the database and not the actual system.
However hooks still run which can leave files in place or run commands
you may not want.

The hooks being run also means `fakeroot pacman -S --dbpath test/ foo --dbonly`
fails because alpm tries to chroot for hooks which requires real root.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 20:46:57 +10:00
morganamilo
be76f8bf06 libalpm: add ALPM_TRANS_FLAG_NOHOOKS
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 20:46:47 +10:00
morganamilo
625f3d645b libalpm: don't use alpm_pgpkey_t in import question
When constructing an import question we never really used a proper gpg
key. We just zero initialize the key, set the uid and fingerprint, and
sent that to the front end.

Instead lets just give the import question a uid and fingerprint field.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 20:43:16 +10:00
morganamilo
e187aa9b48 libalpm: use else when setting fingerprint
The docs [1] say keyid will always be there, so no need to check if it
exists.

[1] https://www.gnupg.org/documentation/manuals/gpgme/Key-objects.html

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:52:23 +10:00
morganamilo
c5c6633dd1 libalpm: rename __foo tyes to _foo
__foo is reserved in c and should not be used.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:52:23 +10:00
morganamilo
2109de613a libalpm: take alpm_trans_t out of the public API
this type is only used internally by alpm

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:52:23 +10:00
Allan McRae
fbb29b5047 repo-add: add --include-sigs option
Pacman now downloads the signature files for all packages when present in a
repository.  That makes distributing signatures within repository databases
redundant and costly.

Do not distribute the package signature files within the repo databases by
default and add an --include-sigs to revert to the old behaviour.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:52:23 +10:00
Allan McRae
0a6fecd072 Release pacman-6.0.1
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:36:22 +10:00
Allan McRae
d5e2c0a551 Update NEWS for pacman-6.0.1 release
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 19:36:20 +10:00
Allan McRae
70d0b2c4b9 Include az_AZ translation files
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 18:02:39 +10:00
Allan McRae
4c3c0e06e5 Update translations from Transifex
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 16:16:20 +10:00
Daan De Meyer
c7e4f45922 pacman-key: Reduce gpg trustdb checks
Every time we modify gpg's state by signing or revoking a key, gpg
marks the trustdb as stale and rechecks it the next time key_is_lsigned()
or key_is_revoked() is called.

Currently, we alternate calls signing of keys and calling key_is_lsigned()
(idem for revoking) which means that for each key we sign (or revoke), gpg
will check the trustdb once.

To avoid checking the trustb so many times, we can simply do all the
key_is_lsigned() and key_is_revoked() checks upfront. Inbetween read
operations the trustdb is not marked stale and inbetween write operations
the trustdb is also not marked stale. This reduces the amount of trustdb
checks from 50 to 1.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 13:43:26 +10:00
Daan De Meyer
160f5bec8c pacman-key: Add --quiet to a few more gpg invocations
Currently, when running pacman-key --populate, gpg prints the
trustdb check output once for each locally signed and revoked key.
When bootstrapping a new container image, about 50 keys get signed
and revoked which leads to a huge amount of output when running
pacman-key which is repeated 50x.

To avoid overloading the user with gpg output, we add --quiet to the gpg
calls generating the trustdb checking output to silence those calls which
gets rid of the trustdb check output on the terminal.

Signed-off-by: Daan De Meyer <daan.j.demeyer@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 13:34:58 +10:00
Hugo Osvaldo Barrera
5da4af2b5d Delete the "Other Utilities" section
Signed-off-by: Hugo Osvaldo Barrera <hugo@barrera.io>
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 10:34:00 +10:00
Hugo Osvaldo Barrera
cf923e734b Update broken links pointing to git.archlinux.org
All of these links are broken since the recent move to
gitlab.archlinux.org.

A few projects are, apparently, only available on GitHub, so I've linked
to that source (hopefully that's only temporary).

For git-clone URLs, I've opted for the https URLs since those can be
used by anyone -- whereas the ssh URLs require the user to be registered
on the gitlab instance which is not open to the public yet.

Signed-off-by: Hugo Osvaldo Barrera <hugo@barrera.io>
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 10:34:00 +10:00
Charlie Sale
efb714b31c Order downloads by descending max_size
When downloading in parallel, sort by package size so that the larger
packages are queued first to fully leverage parallelism.
Addresses FS#70172

Signed-off-by: Charlie Sale <softwaresale01@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 10:34:00 +10:00
morganamilo
f951282bec pactest: add tests for downloading packages from a cdn
Test for downloads that redirect to some sort of cdn where the
redirected url does not relate to the original filename.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 10:34:00 +10:00
morganamilo
2ec6de96a6 only use effective url for urls containing .db or .pkg
Github and other sites redirect their downloads to a cdn. So the
download http://foo.org/myrepo.db may redirect to something like
https://cdn.foo.org/83749327439.

This then causes pacman to try and download the sig as
https://cdn.foo.org/83749327439.sig which is incorrect. In this case
pacman should append .sig to the original url.

However urls like https://archlinux.org/packages/community/x86_64/0ad/download/
Redirect to the mirror, so .sig has to appended after the redirects and
not before.

So we decide if we should append .sig on the original or effective url
based on if the effective url (minus the query part) has .db or .pkg in it.

Fixes FS#71148

---

v2: move variable decleration to start of block
v3: use dbext instead of db
2021-09-04 10:34:00 +10:00
morganamilo
c0026caab0 libalpm: Give -U downloads a random .part name if needed
archweb's download links all ended in /download. This cause all the temp
files to be named download.part. With parallel downloads this results in
multiple downloads to go to the same temp file and breaks the transaction.

Assign random temporary filenames to downloads from URLs that are either
missing a filename, or if the filename does not contain at least three
hyphens (as a well formed package filename does).

While this approach to determining when to use a temporary filename is
not 100% foolproof, it does keep nice looking download progress bar names
when a proper package filename is given. The only downside of not using
temporary files when provided with a filename  with three or more hyphens
is URLs created specifically to bypass temporary filename usage can not
be downloaded in parallel. We probably do not want to download packages
from such URLs anyway.

Fixes FS#71464

Modified-by: Allan McRae (do not use temporary files for realish URLs)
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-09-04 10:33:51 +10:00
Allan McRae
1c5a56884f libmakepkg: reproducibilty for python packages
Arch Linux has been setting PYTHONHASHSEED=0 to create deterministic
.pyc files.  After a thorough review by the Arch Security Team, setting
this variable was determined not to generated vulnerable .pyc files, as
when the loader loads the .pyc file and unmarshalls it, the internal
runtime will just populate the unordered data structures and use a new
runtime hash for them.

Signed-off-by: Allan McRae <allan@archlinux.org>
2021-08-08 22:49:32 +10:00
Allan McRae
fc7986485c Gitlab CI: add bsdtar to Fedora install
Signed-off-by: Allan McRae <allan@archlinux.org>
2021-08-04 23:19:12 +10:00