Starting the download process, even if there is nothing to actually
download, causes an error when pacman is built without curl and has no
XferCommand defined (like our test suite).
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Currently default_pkg_ops is accessed in two different ways.
There is get_file_pkg_ops (in be_package.c) creating a local once-off
'tweaked' copy. As well as load_pkg_for_entry (be_sync.c) which modifies
in-place and uses default_pkg_ops.
This seems rather misleading and fragile approach. Introduce a helper
for the second use-case so that default_pkg_ops is handled consistently
and essentially remains unchanged throughout.
Signed-off-by: Emil Velikov <emil.l.velikov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Until now callee of ALPM download functionality has been in charge of
payload creation both for the main file (e.g. *.pkg) and for the accompanied
*.sig file. One advantage of such solution is that all payloads are
independent and can be fetched in parallel thus exploiting the maximum
level of download parallelism.
To build *.sig file url we've been using a simple string concatenation:
$requested_url + ".sig". Unfortunately there are cases when it does not
work. For example an archlinux.org "Download From Mirror" link looks like
this https://www.archlinux.org/packages/core/x86_64/bash/download/ and
it gets redirected to some mirror. But if we append ".sig" to the end of
the link url and try to download it then archlinux.org returns 404 error.
To overcome this issue we need to follow redirects for the main payload
first, find the final url and only then append '.sig' suffix.
This implies 2 things:
- the signature payload initialization need to be moved to dload.c
as it is the place where we have access to the resolved url
- *.sig is downloaded serially with the main payload and this reduces
level of parallelism
Move *.sig payload creation to dload.c. Once the main payload is fetched
successfully we check if the callee asked to download the accompanied
signature. If yes - create a new payload and add it to mcurl.
*.sig payload does not use server list of the main payload and thus does
not support mirror failover. *.sig file comes from the same server as
the main payload.
Refactor event loop in curl_multi_download_internal() a bit. Instead of
relying on curl_multi_check_finished_download() to return number of new
payloads we simply rerun the loop iteration one more time to check if
there are any active downloads left.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
All users of _alpm_download() have been refactored to the new API.
It is time to remove the old _alpm_download() functionality now.
This change also removes obsolete SIGPIPE signal handler functionality
(this is a leftover from libfetch days).
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Now when all callers of the old alpm_db_update() function are gone we can
remove this implementation. And then rename alpm_dbs_update() function to
alpm_db_update().
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Multiplexed download requires ability to draw UI for multiple active progress
bars. To implement it we use ANSI codes to move cursor up/down and then
redraw the required progress bar.
`pacman_multibar_ui.active_downloads` field represents the list of active
downloads that correspond to progress bars.
`struct pacman_progress_bar` is a data structure for a progress bar.
In some cases (e.g. database downloads) we want to keep progress bars in order.
In some other cases (package downloads) we want to move completed items to the
top of the screen. Function `multibar_move_completed_up` allows to configure
such behavior.
Per discussion in the maillist we do not want to show download progress for
signature files.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Multiplexed database/files downloads will use multiple progress bars.
The UI logic is quite complicated and printing error messages while
handling multiple progress bars is going to be challenging.
Instead we are going to save all ALPM error messages to a list and flush
it at the end of the download process. Use on_progress variable that
blocks error messages printing.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
This is an equivalent of alpm_db_update but for multiplexed (parallel)
download. The difference is that this function accepts list of
databases to update. And then ALPM internals download it in parallel if
possible.
Add a stub for _alpm_multi_download the function that will do parallel
payloads downloads in the future.
Introduce dload_payload->filepath field that contains url path to the
file we download. It is like fileurl field but does not contain
protocol/server part. The rationale for having this field is that with
the curl multidownload the server retry logic is going to move to a curl
callback. And the callback needs to be able to reconstruct the 'next'
fileurl. One will be able to do it by getting the next server url from
'servers' list and then concat with filepath. Once the 'parallel download'
refactoring is over 'fileurl' field will go away.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
The GOTO_ERR define was added in commit 80ae8014 for use in future commits.
There are plenty of places in the code base it can be used, so convert them.
Signed-off-by: Allan McRae <allan@archlinux.org>
realloc can fail just like the other memory allocation functions. Add a
macro to simplify handling of realloc failures, similar to the already
existing MALLOC, CALLOC, etc.
Replace the existing realloc uses with the new macro, allowing us to
move tedious error handling to the macro. Also, in be_package and
be_sync, this fixes hypothetical memory leaks (and thereafter null
pointer dereferences) in case realloc fails to shrink the allocated
memory.
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
We previously has the maximum database size as 25MB. This was set in the days
before repos had as many packages as they do now, and before we started
distributing files databases. Increase this limit to 128MB.
Signed-off-by: Allan McRae <allan@archlinux.org>
Commit 0994893b0e added the
alpm_pkg_get_{make,check}depends functions but forgot to include
logic for parsing these fields from the database. As a result these
functions will always return an empty list.
This commit adds the parsing logic.
Signed-off-by: morganamilo <morganamilo@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
An empty siglevel does not do any signature verification which is
exactly what we want when compiled without gpg support. This is already
allowed in other parts of the codebase and required for the test suite
to pass when compiled without gpg support.
Fixes: FS#60880
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
"wrong or NULL argument passed" is a useless error for end users.
Fixes FS#60880.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Many of these are pointless (e.g. there is no need to explicitly turn on
spellchecking and language dictionaries for the manpages by default).
The only useful modelines are the ones enforcing the project coding
standards for indentation style (and "maybe" filetype/syntax, but
everything except the asciidoc manpages and makepkg.conf is already
autodetected), and indent style can be applied more easily with
.editorconfig
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
If rehash ever failed with a full hash it would return the old hash
that is already full. get_hash_position would then loop forever
because it would never find an empty bucket.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
If you manage to download a bad database (e.g. an html file when
behind a proxy or with a badly configured webserver), pacman makes
sure you know about it. Here is some example output:
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
error: could not open file /var/lib/pacman/sync/extra.db: Unrecognized archive format
I don't know how many times that gets printed because it goes beyond my scrollback
buffer.
Flag a database that we can "open" and "fstat" but not read from as invalid to avoid
this.
Signed-off-by: Allan McRae <allan@archlinux.org>
Many bitfield variables are declared to be enums, because they are
generated using bitwise operations on enums such. However, their
actual values aren't necessary members of their parent enum, so
declaring them 'int' is more accurate.
Signed-off-by: Ivy Foster <ivy.foster@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
This allows functions which return an _alpm_errno_t to always return a
genuine _alpm_errno_t for consistency, even in cases where there are
no errors. Since ALPM_ERR_OK = 0, their callers can still simply check
'err = some_fn(); if (!err) { ... }'.
Signed-off-by: Ivy Foster <ivy.foster@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Making it size_t matches the return value of alpm_list_count() and
avoids the implicit cast to int.
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Since the number of packages is not used anywhere, just return a
boolean to avoid the implicit cast from size_t to int in be_local.c.
Use 0 as success to be consistent with db_validate.
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Signature downloading and DB validation was being based on the most
recent download status for the DB. If a DB successfully downloaded but
a signature did not, db_update would move to the next server. If the
next server tried does not have a more recent copy of the DB, db_update
would not download the DB again and would forget that the DB had
previously been updated. In this case it would skip validation
entirely, leaving an updated DB with the original validation status.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
If given size 0 POSIX allows realloc to return a pointer that is not
suitable for use.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
When a database and its signature is updated non-atomically on a server,
there is a window where a user may update the database but grab the old
signature. The database is marked as invalid by libalpm, which can be
fixed by forcing a refresh (assuming the server has caught up and the
user realizes what has happened) or with a future update of the repo.
Work around this by forcing a repository refresh whenever a database is
invalid.
Signed-off-by: Sebastian Lackner <sebastian@fds-team.de>
Signed-off-by: Allan McRae <allan@archlinux.org>
If a sync database contains a "files" file, the file list will be read.
Currently, there is no known demand for the file list to be lazy loaded by
any libalpm frontend, so these files are read whenever present. Lazy loading
can be implemented when a demand exists.
Signed-off-by: Allan McRae <allan@archlinux.org>
This allows frontends to select between the .db and .files databases
currently supplied by repo-add or any other compatible database.
Signed-off-by: Allan McRae <allan@archlinux.org>
The change in commit 9d96bed9d6 causes download errors for the .db.sig file
in case the final URL for the .db file contains query strings or other
unexpected stuff. This commit isn't intended to be a total solution, but it
should eliminate the problem in the most obvious cases.
Signed-off-by: Allan McRae <allan@archlinux.org>
This will allow pacman to parse its config file in a single pass and
removes the need for the *_SET siglevels in alpm that were only required
for pacman's siglevel inheritance.
Signed-off-by: Andrew Gregory <andrew.gregory.8@gmail.com>
This commit adds support to libalpm to parse the pkgbase present in
packages .PKGINFO files, writing the PKGBASE to the %BASE% section of
the local DBs desc files and for parsing it again when loading the local
DB
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Commit 9d96bed9 attempts to use the same effective URL for the db and its
signature download. However, this information is not available when we use
an external downloader, resulting in a crash.
Fall back to the old method when the effective URL is unavailable.
Signed-off-by: Allan McRae <allan@archlinux.org>
If download server is dynamic mirror chances are that db file download
and db file signature download are redirected to different mirrors,
resulting in invalid signature.
This uses effective URL for db file signature download and makes the
files always match.
Signed-off-by: Christian Hesse <mail@eworm.de>
Signed-off-by: Allan McRae <allan@archlinux.org>
