Mercurial > ~samis > hgweb.cgi > blag

annotate posts/Gentoo_Repositories.md @ 43:2227cf67c70f

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Create tag indexes for ikiwiki creating tag page tags/shell *** creating tag page tags/netbsd *** creating tag page tags/draft *** creating tag page tags/sdf
author Samuel Hodgkins <samuel.hodgkins@sky.com>
date Sun, 25 Mar 2018 04:43:53 +0000
parents 197e15796e63
children a7f907874a07
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
1 [[!meta title="Using eselect-repository with Gentoo" author="samis"]]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
2 [[!tag tech linux gentoo portage]]
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
3 # Background
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
4 Some time after installing Gentoo, I moved away from syncing the portage tree via Git, because having history of the repository is nice and as a bonus, the repository can be synchronized from GitHub over HTTPS while rsync uses plain text.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
5 Commits in the repository are also GPG-signed, but the current version of portage does not use this to validate the repository [(yet)](https://github.com/gentoo/portage/commit/7a507942b669b6a157612e8f7ce3fe362c39b38f)
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
6
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
7 However, the official Gentoo package repository is not enough, and even though I use [Bedrock Linux](https://bedrocklinux.org/), it's nice to have packages designed for Gentoo so I use a number of unofficial repositories or overlays to add more packages and the conventional tool to use is [Layman](https://wiki.gentoo.org/wiki/Layman) which has a decent-enough interface and support for a wide variety of methods to sync an overlay - all of the commonly-known OSS version control systems and some others.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
8 These overlays are updated at the same time as the official repository via portage (which calls down to layman to do the actual sync).
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
9
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
10 When I switched to syncing the tree via Git, I used a repository that had pre-generated all the additional information not directly contained in the actual 'gentoo' repository like metadata and other files.
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
11 This has the advantage of being simpler to use, but as a consequence the repository's history and disk-usage is bloated because it has to track the entire history of not only the ebuilds and related files, but also the easily-reproducible metadata and files taken from other repos - and recently I found a package that resolves the problem with ease-of-use so I decided to try it out (I had other reasons, but that's for later)
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
12
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
13 At the same time, I thought that since most of my used overlays were using git, wouldn't it make sense to sync them the same way, rather than going through layman?
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
14 It'd certainly be simpler, a quick look showed that there was a package that provided a good interface for this, [eselect-repository](https://github.com/mgorny/eselect-repository) that also was much simpler than layman, being under 500 total lines in a file compared to layman's roughly 10kloc (estimated by cloc after removing autoconf and doc).
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
15 Any non-git overlays would stay managed by layman, though.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
16
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
17 # Doing the move
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
18 (Note: Some parts left out or tweaked for the purposes of this post.)
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
19
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
20 1. I added the mv overlay to add the package with the scripts, and then installed both it and `eselect-repository` with emerge.
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
21 2. Ensure `/etc/portage/repos.conf/gentoo.conf`looks like this (change the clone-depth if you don't want a full clone):
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
22
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
23 [DEFAULT]
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
24 main-repo = gentoo
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
25
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
26 [gentoo]
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
27 location = /usr/portage
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
28 sync-type = git
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
29 sync-uri = https://github.com/gentoo/gentoo.git
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
30 auto-sync = yes
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
31 clone-depth = 0
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
32
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
33 3. Tweak `/etc/eselect/repository.conf` so it uses the original sources, rather than the ones with metadata in the history.
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
34 4. Make a note of your layman-managed git overlays somewhere to re-add them shortly afterwards.
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
35 5. Remove (or disable) all the git overlays using `layman` (`-d` or `-D`)
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
36 6. Re-add the overlays using `eselect repository enable` for overlays on the main list, and `add` for those not
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
37 7. Run your first sync command. Ignore the messages about non-existent directories, they'll go away as each overlay is created when synced.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
38
39
f49c9ba2f8e3 Correct and alter draft
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 36
diff changeset
39 # Unrelated Benefits
36
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
40 While this section isn't related to either eselect-repository or layman, it is related to the changes and why I did them. Essentially, after finding out about a certain overlay which had a large number of packages, I noticed that eix-sync was taking a good amount of time to update the database for this large overlay - looking at another, I noticed that I never generated the metadata cache for any of my overlays. While googling about this, I found the complete set of shell scripts to update this for all overlays and repositories, along with other things (and also found a post pointing out the wastefulness of storing all the metadata in git for the Gentoo repository). The end-result was this post, and the generated metadata cache, while taking up-front CPU time, makes the eix updates faster overall and hopefully will also provide a nice speed boost for portage in general.
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
41
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
42 # Next?
84e17c0f39c2 Initial draft of new Gentoo blog post
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents:
diff changeset
43 While there are not many next steps, the most obvious one is to add support in portage for the sync methods that Layman provides but Portage itself does not. Even though I'm not an expert in Python, this shouldn't be difficult to implement.
40
197e15796e63 Add note about pre-existing mercurial sync module as patch
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 39
diff changeset
44
197e15796e63 Add note about pre-existing mercurial sync module as patch
Samuel Hodgkins <samuel.hodgkins@sky.com>
parents: 39
diff changeset
45 (Addendum: Someone in fact already wrote a module for Mercurial syncing, which is one of the two non-git overlay types I use. You can find it on Gentoo's bugzilla and apply it as a user patch, it works perfectly fine in my experience.)