The main payload final name might be affected by url redirects or
Content-Disposition HTTP header value.
We want to make sure that accompanion *.sig filename always matches the
package filename. So ignore finalname/Content-Disposition for the *.sig file.
It also helps to fix a corner case when the download URL does not contain
a filename and server provides Content-Disposition for the main payload
but not for the signature payload.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Pacman has a 'key in keyring' verification step that makes sure the signatures
have a valid keyid. Currently pacman parses embedded package signatures only.
Add a fallback to detached signatures. If embedded signature is missing then it
tries to read corresponding *.sig file and get keyid from there.
Verification:
debug: found cached pkg: /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst
debug: found detached signature /var/cache/pacman/pkg/glib-networking-2.64.3-1-x86_64.pkg.tar.zst.sig with size 310
debug: found signature key: A5E9288C4FA415FA
debug: looking up key A5E9288C4FA415FA locally
debug: key lookup success, key exists
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
In case if *.pkg exists but *.sig file does not we still have to pass
the pkg to multi_download API.
To avoid redownloading *.pkg file we use CURLOPT_TIMECONDITION curl option.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
It is similar to _alpm_filecache_find() but does not return a
dynamically allocated memory to user. Thus the user does not need to
free this resource.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Current code uses an incrementing counter to check whether a function
returned error:
errors += some_function();
if(errors) { goto finish }
Replace with a more standard variable
errors = some_function();
if(errors) { goto finish }
Rename 'errors' variable to a more typical 'ret'.
Avoid reporting both ALPM_EVENT_PKG_RETRIEVE_FAILED and
ALPM_EVENT_PKG_RETRIEVE_DONE in the error path.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Until now callee of ALPM download functionality has been in charge of
payload creation both for the main file (e.g. *.pkg) and for the accompanied
*.sig file. One advantage of such solution is that all payloads are
independent and can be fetched in parallel thus exploiting the maximum
level of download parallelism.
To build *.sig file url we've been using a simple string concatenation:
$requested_url + ".sig". Unfortunately there are cases when it does not
work. For example an archlinux.org "Download From Mirror" link looks like
this https://www.archlinux.org/packages/core/x86_64/bash/download/ and
it gets redirected to some mirror. But if we append ".sig" to the end of
the link url and try to download it then archlinux.org returns 404 error.
To overcome this issue we need to follow redirects for the main payload
first, find the final url and only then append '.sig' suffix.
This implies 2 things:
- the signature payload initialization need to be moved to dload.c
as it is the place where we have access to the resolved url
- *.sig is downloaded serially with the main payload and this reduces
level of parallelism
Move *.sig payload creation to dload.c. Once the main payload is fetched
successfully we check if the callee asked to download the accompanied
signature. If yes - create a new payload and add it to mcurl.
*.sig payload does not use server list of the main payload and thus does
not support mirror failover. *.sig file comes from the same server as
the main payload.
Refactor event loop in curl_multi_download_internal() a bit. Instead of
relying on curl_multi_check_finished_download() to return number of new
payloads we simply rerun the loop iteration one more time to check if
there are any active downloads left.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
All users of _alpm_download() have been refactored to the new API.
It is time to remove the old _alpm_download() functionality now.
This change also removes obsolete SIGPIPE signal handler functionality
(this is a leftover from libfetch days).
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Installing remote packages using its URL is an interesting case for ALPM
API. Unlike package sync ('pacman -S pkg1 pkg2') '-U' does not deal with
server mirror list. Thus _alpm_multi_download() should be able to
handle file download for payloads that either have 'fileurl' field
or pair of fields ('servers' and 'filepath') set.
Signature for alpm_fetch_pkgurl() has changed and it accepts an
output list that is populated with filepaths to fetched packages.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
In order to use gettext on systems where it is not part of libc, the
correct linker flags are needed in libalpm.pc (for static compilation).
This has never been the case.
The new meson build system currently only checks for ngettext in libc,
but does not fall back to searching for the existence of -lintl; add it
to the libalpm dependencies.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
Given RFC 4880 provides the code to do this calculation, I am not sure
how I managed to stuff that up! This bug was only exposed when a
signature made with "include-key-block" was added to the Arch repos,
which provided a subpacket with the required size to hit this issue.
Signed-off-by: Allan McRae <allan@archlinux.org>
Now when all callers of the old alpm_db_update() function are gone we can
remove this implementation. And then rename alpm_dbs_update() function to
alpm_db_update().
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Create a list of dload_payloads and pass it to the new _alpm_multi_*
interface.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Multiplexed download requires ability to draw UI for multiple active progress
bars. To implement it we use ANSI codes to move cursor up/down and then
redraw the required progress bar.
`pacman_multibar_ui.active_downloads` field represents the list of active
downloads that correspond to progress bars.
`struct pacman_progress_bar` is a data structure for a progress bar.
In some cases (e.g. database downloads) we want to keep progress bars in order.
In some other cases (package downloads) we want to move completed items to the
top of the screen. Function `multibar_move_completed_up` allows to configure
such behavior.
Per discussion in the maillist we do not want to show download progress for
signature files.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
With the previous download interface the callback uses the first progress
event as 'download has started' signal. Unfortunately it does not work with
up-to-date files that never receive 'download progress' events.
Up-to-date database messages are currently handled in sync_syncdbs()
after the sequential download is completed and a result from ALPM is
received. But this is not going to work with multiplexed download
interface that returns the result only after all files are completed.
Another problem with 'first progress event is the beginning of the
download' is that such events time are unpredictable. Thus the UI progress
bar order might differ from what has been passed by client to
alpm_dbs_update() function. We actually want to keep the dbs progress bars
in a strict order.
To help to solve the given problems extend the download callback to
allow 2 more events - download started and completed. 'Download started'
events appear in the same order as in the list given by a client.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Multiplexed database/files downloads will use multiple progress bars.
The UI logic is quite complicated and printing error messages while
handling multiple progress bars is going to be challenging.
Instead we are going to save all ALPM error messages to a list and flush
it at the end of the download process. Use on_progress variable that
blocks error messages printing.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
curl_multi_download_internal() is the main loop that creates up to
'ParallelDownloads' easy curl handles, adds them to mcurl and then
performs curl execution. This is when the paralled downloads happens.
Once any of the downloads complete the function checks its result.
In case if the download fails it initiates retry with the next server
from payload->servers list. At the download completion all the payload
resources are cleaned up.
curl_multi_check_finished_download() is essentially refactored version of
curl_download_internal() adopted for multi_curl. Once mcurl porting is
complete curl_download_internal() will be removed.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
It is an equivalent of _alpm_download but accepts a list of payloads.
curl_multi_download_internal() is a stub at this moment and will be
implemented in the later commits of this patch series.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
dload_payload->curlerr is a field that is used inside
curl_download_internal() function only. It can be converted to a local
variable.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
To be able to run multiple download in parallel efficiently we need to
use curl_multi interface [1]. It introduces a set of APIs over new type
of handler 'CURLM'.
Create CURLM object at the application start and set it to global ALPM
context.
The 'single-download' CURL handle moves to payload struct. A new CURL
handle is created for each payload with intention to be processed by CURLM.
Note that curl_download_internal() is not ported to CURLM interface due
to the fact that the function will go away soon.
[1] https://curl.haxx.se/libcurl/c/libcurl-multi.html
Signed-off-by: Allan McRae <allan@archlinux.org>
This is an equivalent of alpm_db_update but for multiplexed (parallel)
download. The difference is that this function accepts list of
databases to update. And then ALPM internals download it in parallel if
possible.
Add a stub for _alpm_multi_download the function that will do parallel
payloads downloads in the future.
Introduce dload_payload->filepath field that contains url path to the
file we download. It is like fileurl field but does not contain
protocol/server part. The rationale for having this field is that with
the curl multidownload the server retry logic is going to move to a curl
callback. And the callback needs to be able to reconstruct the 'next'
fileurl. One will be able to do it by getting the next server url from
'servers' list and then concat with filepath. Once the 'parallel download'
refactoring is over 'fileurl' field will go away.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
It includes pacman.conf new 'ParallelDownloads' option that
specifies how many concurrent downloads cURL starts in parallel.
Add alpm_option_set_parallel_downloads() ALPM function that
allows to set this config option programmatically.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
The GOTO_ERR define was added in commit 80ae8014 for use in future commits.
There are plenty of places in the code base it can be used, so convert them.
Signed-off-by: Allan McRae <allan@archlinux.org>
Use STRDUP() over strdup() to catch memory allocation errors.
There are still some instances of strdup left, but these are in functions
that currently have no error path and would require a larger rework.
Signed-off-by: Allan McRae <allan@archlinux.org>
realloc can fail just like the other memory allocation functions. Add a
macro to simplify handling of realloc failures, similar to the already
existing MALLOC, CALLOC, etc.
Replace the existing realloc uses with the new macro, allowing us to
move tedious error handling to the macro. Also, in be_package and
be_sync, this fixes hypothetical memory leaks (and thereafter null
pointer dereferences) in case realloc fails to shrink the allocated
memory.
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Following the example of the recently added GOTO_ERR, adding the file and
line number in addition to the function name in our debug messages is
potentially useful.
Signed-off-by: Allan McRae <allan@archlinux.org>
This is a macro similar to RET_ERR but useful in the case when we need
to record an error and then jump to some cleanup section.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Since commit 2ee7a8d8, there is no cleanup needed in this function. Just
return instead of jumping to the cleanup label.
Signed-off-by: Allan McRae <allan@archlinux.org>
One reason why the function returns an error is some repo
does not have any servers.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Currently, download_files() creates payloads for all packages then
iterates over them, calling download_single_file. This can be
simplified by looping over packages and constructing the payload as needed.
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
Current flow looks like
loop dbs_sync {
loop pkgs {
if pkg.db == db then process(pkg, db)
}
}
Package sync transaction always has a counterpart in the dbs_sync list
(I cannot come up with a use-case when it is not true). So the loop can
be simplified to:
loop pkgs {
process(pkg, pkg.db)
}
Tested: 'ninja test' & manually by using pacman with this patch for a
week
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
It looks like this function has never actually worked. The current list
is never set to NULL after being freed. So the new deps were just
appended to the already freed list, leading to a segfault.
Signed-off-by: Allan McRae <allan@archlinux.org>
Commit e6a6d307 detected complete part files by comparing a payload's
max_size to initial_size. However, these values are also equal when we
use pacman -U on a URL as max_size is set to 0 in that case. Add a further
condition to avoid that.
Signed-off-by: Allan McRae <allan@archlinux.org>
We previously has the maximum database size as 25MB. This was set in the days
before repos had as many packages as they do now, and before we started
distributing files databases. Increase this limit to 128MB.
Signed-off-by: Allan McRae <allan@archlinux.org>
The documentation of the return types of alpm_mtree_next was incorrect.
This extended into the relevant function in be_local.c.
Also, return explicit integer values, rather than the ARCHIVE_xxx values,
to avoid unnecessarily exposing frontends to libarchive internals (even
though it makes no functional difference).
Original-work-by: morganamilo <morganamilo@archlinux.org>
Signed-off-by: Allan McRae <allan@archlinux.org>
In rare cases, likely due to a well timed Ctrl+C, but possibly due to a
broken mirror, a ".part" file may have size at least that of the correct
package size.
When encountering this issue, currently pacman fails in different ways
depending on where the package falls in the list to download. If last,
"wrong or NULL argument passed" error is reported, or a "invalid or
corrupt package" issue if not.
Capture these .part files, and remove the extension. This lets pacman
either use the package if valid, or offer to remove it if it fails checksum
or signature verification.
Signed-off-by: Allan McRae <allan@archlinux.org>
We now store key structs of our missing key info, so can not search the list
for string matches. This caused missing keys to be downloaded once for every
package they signed.
Signed-off-by: Allan McRae <allan@archlinux.org>
Previously, pacman treated no matches and an error during search the
same.
To fix this, alpm_db_search now returns its status as an int and
instead takes the to be returned list as a param. Allowing front ends to
easily differentiate between errors and no matches.
Signed-off-by: morganamilo <morganamilo@gmail.com>
Signed-off-by: Allan McRae <allan@archlinux.org>
This allows pacman to print the correct error message when checking keys
and libalpm has been compiled without gpgme support.
Signed-off-by: Allan McRae <allan@archlinux.org>
The dummy checksigs function never sets count to 0, leaving it
unitialized. This caused the siglist cleanup to try and free the empty
list.
Signed-off-by: Allan McRae <allan@archlinux.org>
