This commit adds an iterator interface for reading files from the
syncdbs. Instead of using alpm_pkg_get_files(), you now get the files
from the database using alpm_db_files_open(), you then use
alpm_db_files_next() to iterate through the files for each package. If
you want to actually load the files from that package you then use
alpm_db_files_load().
This means alpm_pkg_get_files() will always return empty for syncdbs,
even on .files databases, however these functions still work on the
localdb and loaded packages.
This aproach is faster when dumping the entire file list but slower when
searching for a specific package.
The memory usage of pacman is drastically less. See below.
build/pacman -Fl 0.55s user 0.01s system 99% cpu 0.556 total
build/pacman -Fl pacman 0.46s user 0.01s system 99% cpu 0.472 total
build/pacman -Fx pacman 2.88s user 0.09s system 99% cpu 2.965 total
pacman -Fl 1.60s user 0.13s system 99% cpu 1.731 total
pacman -Fl pacman 0.24s user 0.04s system 99% cpu 0.283 total
pacman -Fx pacman 2.45s user 0.14s system 99% cpu 2.593 total
Peak Memory
build/pacman -Fl 43.52MB
build/pacman -Fl pacmam 11.292MB
pacman -Fl 677.048MB
pacman -Fl pacman 163.288MB
When constructing an import question we never really used a proper gpg
key. We just zero initialize the key, set the uid and fingerprint, and
sent that to the front end.
Instead lets just give the import question a uid and fingerprint field.
Signed-off-by: Allan McRae <allan@archlinux.org>
This partially fixes FS#67850
It fixes the case for -S'ing packages but not -U'ing urls.
pacman -S a/a b/b
resolving dependencies...
error: packages a and b have the same filename: a-1-1-any.pkg.tar.zst
error: failed to prepare transaction (duplicate filename)
Signed-off-by: Allan McRae <allan@archlinux.org>
When a download fails on one mirror a new download is started on the
next mirror. This causes the ammount downloaded to reset, confusing the
rate math and making it display a negative rate.
This is further complicated by the fact that a download may be resumed
from where it is or started over.
To account for this we alert the frontend that the download was
restarted. Pacman then starts the progress bar over.
Signed-off-by: Allan McRae <allan@archlinux.org>
Every alpm_option_set function clones the input so lets be more
consistent. Also this fixes servers not being sanatized.
Signed-off-by: Allan McRae <allan@archlinux.org>
Restore the prior indicator whether or not databases were up to date.
0 is used to indicate if *any* db was actually updated as callers are
more likely to care about that than if *all* dbs were updated.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Our callbacks require front-ends to maintain state in order to provide
reasonable output. The new download callback in particular requires
much more complex state information to be saved. Without the ability to
provide context, state must be saved globally, which may not be possible
for all front-ends. Scripting language bindings in particular have no
way to register per-handle callbacks without some form of context.
Implements: FS#12721
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
This allows architecture to be multivalued. On x86-64 machines, this
could be something like:
Architecture = x86-64-v3 x86-64
We use the first specified Architecture value in mirrorlist $arch
variable replacement, as this is backwards-compatible and sane.
Original-patch-by: Dan McGee <dan@archlinux.org>
Patch-updated-by: Allan McRae <allan@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
Total download callback called right before packages start downloaded.
But we already have an event for such event (ALPM_EVENT_PKG_RETRIEVE_START)
and it is naturally to use the event to pass information about expected
download size.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Also change the group's title to point to the group's man page.
This makes generated man pages be named libalpm_* which is more
consistent with what library man pages are usually called.
Signed-off-by: Allan McRae <allan@archlinux.org>
The comment makes it seem that the result itself is an error code. But
all it does is simply return -1 to indicate an error occured;
Signed-off-by: Allan McRae <allan@archlinux.org>
Pacman has a 'key in keyring' verification step that makes sure the signatures
have a valid keyid. Currently pacman parses embedded package signatures only.
Add a fallback to detached signatures. If embedded signature is missing then it
tries to read corresponding *.sig file and get keyid from there.
Verification:
debug: found cached pkg: /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst
debug: found detached signature /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst.sig with size 310
debug: found signature key: A5E9288C4FA415FA
debug: looking up key A5E9288C4FA415FA locally
debug: key lookup success, key exists
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Installing remote packages using its URL is an interesting case for ALPM
API. Unlike package sync ('pacman -S pkg1 pkg2') '-U' does not deal with
server mirror list. Thus _alpm_multi_download() should be able to
handle file download for payloads that either have 'fileurl' field
or pair of fields ('servers' and 'filepath') set.
Signature for alpm_fetch_pkgurl() has changed and it accepts an
output list that is populated with filepaths to fetched packages.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Now when all callers of the old alpm_db_update() function are gone we can
remove this implementation. And then rename alpm_dbs_update() function to
alpm_db_update().
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Create a list of dload_payloads and pass it to the new _alpm_multi_*
interface.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
With the previous download interface the callback uses the first progress
event as 'download has started' signal. Unfortunately it does not work with
up-to-date files that never receive 'download progress' events.
Up-to-date database messages are currently handled in sync_syncdbs()
after the sequential download is completed and a result from ALPM is
received. But this is not going to work with multiplexed download
interface that returns the result only after all files are completed.
Another problem with 'first progress event is the beginning of the
download' is that such events time are unpredictable. Thus the UI progress
bar order might differ from what has been passed by client to
alpm_dbs_update() function. We actually want to keep the dbs progress bars
in a strict order.
To help to solve the given problems extend the download callback to
allow 2 more events - download started and completed. 'Download started'
events appear in the same order as in the list given by a client.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Multiplexed database/files downloads will use multiple progress bars.
The UI logic is quite complicated and printing error messages while
handling multiple progress bars is going to be challenging.
Instead we are going to save all ALPM error messages to a list and flush
it at the end of the download process. Use on_progress variable that
blocks error messages printing.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
This is an equivalent of alpm_db_update but for multiplexed (parallel)
download. The difference is that this function accepts list of
databases to update. And then ALPM internals download it in parallel if
possible.
Add a stub for _alpm_multi_download the function that will do parallel
payloads downloads in the future.
Introduce dload_payload->filepath field that contains url path to the
file we download. It is like fileurl field but does not contain
protocol/server part. The rationale for having this field is that with
the curl multidownload the server retry logic is going to move to a curl
callback. And the callback needs to be able to reconstruct the 'next'
fileurl. One will be able to do it by getting the next server url from
'servers' list and then concat with filepath. Once the 'parallel download'
refactoring is over 'fileurl' field will go away.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
It includes pacman.conf new 'ParallelDownloads' option that
specifies how many concurrent downloads cURL starts in parallel.
Add alpm_option_set_parallel_downloads() ALPM function that
allows to set this config option programmatically.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
The documentation of the return types of alpm_mtree_next was incorrect.
This extended into the relevant function in be_local.c.
Also, return explicit integer values, rather than the ARCHIVE_xxx values,
to avoid unnecessarily exposing frontends to libarchive internals (even
though it makes no functional difference).
Original-work-by: morganamilo <morganamilo@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
