Mirrors #38
Labels
No Label
blocked upstream
bug
build-failure
duplicate
enhancement
help wanted
informational
invalid
invalid/corrupt package
packaging issue
priority: high
question
support
wontfix
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: ALHP/ALHP.GO#38
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Providing a rsync URL will allow creating new mirrors of the repo. I may provide one of them in the future.
I would like to host a mirror too if possible!
I'll see if I get that setup later today.
Thanks! I'm looking forward to it.
Mirroring via rsync
source:
rsync://alhp.harting.dev/alhplastupdate:
https://alhp.harting.dev/lastupdateexpected size: around 30GB per
march(e.g. x86-64-v3) you want to sync.I highly recommend using this script from Archlinux as base for a sync service. It checks if a sync needs to be done (by comparing lastupdate before syncing). It's also worth to read comments.
You are free to mirror only one march or exclude
/logs, see line 90 of said script above.If you have setup a mirror, let me know here in this issue (if possible with the location of the mirror; country/region is enough), then I'll add your mirror to the README for users to choose from.
ooo Thank you so much! I'll set this up ASAP
Here's my mirror:
https://www.gardling.com/alhpit syncs every 5 minutes (I did some modifications to the script you referenced).If more people start mirroring this. Maybe you can setup a
ahlp-mirrorlistpackage just like thechaoticaurunofficial repo with theirchaoticaur-mirrorlistpackage. Just a thought 👍I've had been thinking, I don't know if the build server is the same as the repo (I presume so), if we had two mirrors, we could use primarily those mirrors and offload the build server.
I don't know if it is really useful but what your thoughts?
I wouldn't be able to host a x86-64-v3 build server, but I can host a x86-64-v2 one (like I mentioned here: https://git.harting.dev/anonfunc/ALHP.GO/issues/17#issuecomment-746). But yea, that is a good idea. That other server would have to be run by a trusted person as they're also now compiling the packages themselves too, so those packages can't just be verified by a checksum or something.
But I don't know how that would be organized.
No really what I've thought, let me try to explain it again.
At the moment the Alhp server is the build server and repo at the same time.
My thoughts is, the current Alhp server, just become the build server, then the repos mirror would retrieve the build package and distribute it.
I don't know if it is more clear that way
Oh ok, I get what you mean. Like kinda how the official arch repos are structured, right?
Yup! Just like the arch repos structure, sorry for my bad explanation it just super late where I live so ^^'
Ah ok. Yea I like that idea!
Probably not a bad move. I also want to bring the keyring into the AUR to prevent people getting trust or other stuff wrong, so a mirrorlist would not hurt either.
I want to move ALHP to a dedicated server if it ever reaches enough support. Sadly there is no way to contribute such a server directly, since trust issues would be unavoidable.
Yup that a big problem,and there is the fact that some mirrors could end-up like some outdated server on arch
If I wanted to setup something like this (a mirror) would there be a way to limit the amount of traffic I get? (without throwing annoying errors at users)?
Without trowing an error no, but you can limit the number of connections/bandwith
You can always rate-limit your webserver. Here and here is the doc for nginx, there should be something similar available for apache.
I see, what kind of traffic can I expect? 100 gb per month? 1000?
Difficult to answer, since it really depends on many factors. I have no dedicated stats for ALHP, but considering the whole server traffic usage I would imagine <500GB/month.
But, like I said, it really depends on a lot of factors, like overall users, your mirror location, etc.
TL;DR: Don't mirror if you're on a tight traffic budget.
Thank you very much for your response! I hope I can assist so I can alleviate some of the traffic to the main server. Just need to discuss it with my hosting company.
Provide a rsync URL for mirrorsto MirrorsInspired by https://cloudflaremirrors.com/archlinux I've set up an experimental mirror: https://alhp.krautflare.de/
Usage:
Server = https://alhp.krautflare.de/$repo/os/$arch/Backend server is only 400 Mbit/s (DE) but that should not matter. It is using cloudflare extensively (all their meme stuff like tiered caching, tunneling and whatnot). Means, the more people use it, the more content gets cached by cf and the faster it will be (hopefully).
I sync every 20m and it is currently configured to cache
*.(zst(.sig)?)for 1 month and for any other files (this includes the directory listing) 20 min.The only thing which won't work is if a package is rebuilt with the same version number / file name - the old version will be stuck in cf's cache. Question: Does this happen? If yes, how often? In case this occasionally happens, I could write a script which nukes them from the cf cache via their API after each rsync.The cache nuke via API thing is now implemented and working (comparing ctimes of files to the last run).Feedback and suggestions welcome, and thanks for this project @anonfunc
Sounds good. I'm almost done creating an alhp-{keyring,mirrorlist} package for ALHP, so I'm going to include that mirror in there (probably going to upload it soon).
That is one issue I still like to tackle. ALHP should increase its buildversion (so package-X.Y-Z.1, the added part behind pgkrel) if a rebuild of the same version was done. I haven't had a chance to implement that (yet), but it's on the todo.Regarding how often that happens currently: Overall it's a rare occurrence, but it does happen (as with
nextcloud-clientorlinux-ltsjust recently).ALHP now increases its build-version correctly.
Both packages are now online on the AUR.
https://aur.archlinux.org/packages/alhp-mirrorlist/
https://aur.archlinux.org/packages/alhp-keyring/
The
alhp-keyringpackage replaces manual key management, whilealhp-mirrorlistprovides an up-to-date selection of mirrors to use.I'm going to edit the README shortly.
Besides this I moved all ALHP repos into a new organization ALHP, as you may have noticed.
The alhp-mirrorlist repo contains the mirrorlist used in the AUR package. All issues with mirrors and new-mirror-PRs should be opened there.
One thing that i think should be done, is to provide
alhp-keyringin the repo, it could be added in the core repo aside of thearchlinux-keyring.The reason is that not everybody always update they're aur packages, and since the keyring is more crucial than the mirrorlist, it could prevent some frustration.
That should be possible.
@incognico What's the server software you are currently using to host the files? Asking this because I tried two times to download a large package from your mirror with 5 concurrent connections and it killed the server process (I guess?) both times, making Cloudflare respond with timeout (524) errors.
Maybe using Nginx would be a better choice for resource managing while serving files (Just like ALHP's main server)?
@alissonlauffer Technically you are downloading form cloudflare servers, so the endpoint for the concurrent connections should be cloudflare (which fetches with plain http1 from my server). Maybe it was a coincidence as I was upgrading cloudflared yesterday. Could you test again and/or post me your XferCommand (or the command you have used) so I can try to reproduce it?