|
Home | Uniquely NZ | Travel |
|
| Using Git for Software Development Building and Patching Ubuntu Kernels and Cinnamon |
Patching an Ubuntu Linux Kernel
The first part of this article has been written in response to a requirement to patch a new xhci (USB) driver into the kernel for my Helios Ultrabook if only to make sure that a particular problem was understood. It is therefore not comprehensive and adopts simple solutions to a particular problem rather than an approach which is very flexible. I am not looking to patching the mainline kernels at kernel.org but those which have been adapted and developed for Debian/Ubuntu and thus for Linux Mint. In general these have a longer life and have more uppstream changes backported into them than the original kernels they are based on but arguably this may lead to less continuity between kernels with changes backported to sat wily and trusty but not to vivid. It also leads to some confusion as to patch say the 3.19 kernel used by Mint 17.3 you need to work on ubuntu-vivid and for 4.2 one works on ubuntu-wily. NOTE: The patch used here as an example has been applied from kernel 4.2.0-28.33 and there are many more recent kernels available now for Mint 17.3 and Mint 18 should not show th problem.
The current Ubuntu kernels are at http://kernel.ubuntu.com/git/?ofs=750 and there are basic instructions at https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel which I started off using. It is quite a good starting point but I found that I had to do a few extra activities. I have tried to break the activities down into several parts:
Git Installation and Configuration if you are using it for the first time.
You may already have Git installed, if not the following is my standard installation and configuration
sudo apt-get install git gitk leafpad meld
git config --global user.name "your_name_or_your_username_on_github"
git config --global user.email "yourname@your_email"
git config --global core.editor leafpad
git config --global diff.tool meld
git config --global merge.tool meld
git config --global color.ui auto
git config --global push.default simple
The above not only installs Git but its visualiser gitk, a merging program meld and a simple editor leafpad which will only be used by Git.
The first two pieces of configuration are to provide a suitable name and email which is added to every commit you make. If you intend to use github there are advantages to having then the same and using your github username rather than your full name.
The use of a simple text editor called leafpad avoids all sorts of problems in using the same editor for your editing of files and within git. meld is a not only a really good difference tool and is also set to be the default for hand crafting conflicts in a merge. The color.ui auto allows suitable terminal programs to display some information in colour. The push.default simple is to make sure that you only push your master, not branches, to the remote repository by default - it should not be needed with the latest versions of Git but I started before that was the default so some of my early repositories seemed to have a different setting.
Everything in the configuration with --global is available throughout your machine but needs to be redone on other machines. the command git config --list shows your current configuration. Run it outside of a working directory to see just the global settings you have made.
Install software required to build Linux Kernels
The following are required to build kernels
sudo apt-get install fakeroot kernel-wedge build-esssential dpkg
Under Mint 18 I found I could not install build-essential from the command line until I had added the Ubuntu main repository by:
sudo add-apt-repository main
And I then had to install it by using the Synaptic Package Manager remembering to Reload first. I do not understand why the usual apt-get install did not work, it must be to do with Mint's priorities.
build-essential seems to just add a list of 'dependences' which you can see make sure it is there and see more about by cat /usr/share/doc/build-essential/list in a terminal.
Set up build environment
If you've not built a kernel on your system before, there are some packages needed and dependencies which need to be satisfied before you can successfully build a kernel. The article I quote just says you can get these installed with:
sudo apt-get build-dep linux-image-$(uname -r)
but this only works if you have first enabled the source code repositories in Ubuntu and Mint. Open Software Sources from the Menu and under Official repositories tick the box under Source Code to enable source code repositories then click Update Cache before the terminal string above. It brings in a huge number of packages (~250 Mbytes).
Install other required packages
sudo apt-get install linux-cloud-tools-common linux-tools-common
These may be needed when you install your compiled kernel later.
Finding and downloading the Kernel Source using Git.
All of the Ubuntu Kernel source is maintained under git. The source for each release is maintained in its own git repository on kernel.ubuntu.com. To obtain a local copy you can simply git clone the repository for the release you are interested in as shown below. For example to obtain the vivid tree which matches Mint 17.3:
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-vivid.git
This put all the files into the folder ubuntu-vivid in your home folder. The Ubuntu Git repository contain many different versions and you can see the overview on the web at http://kernel.ubuntu.com/git/?ofs=750 and the index of the vivid (kernel 3.19) source on git at http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/
Once you have cloned the source from git you can see the individual versions (3.19.0-39 etc) has a tag. You can see the huge list of tags by changing into the folder and doing git tag as below but it is better to use grep to filter out the ones of interest:
cd ubuntu-vivid
git tag | grep -i 3.19.0-3
and you can then get to any earlier kernel version to see or work on by
git checkout tagname
If you want to patch at that point it is best to create a branch even if you delete it afterwards for example if you want to patch the base version of Mint 17.3 which uses kernel 3.19.0-32.37 you would create a branch for the work like this.
git checkout -b working_3.19.0-32.37 Ubuntu-3.19.0-32.37
and after you have finished you have the option of deleting it by
git branch -d working_3.19.0-32.37
If you plan to use git extensively or send any patches back git itself should have a little setup done.
Applying a Patch directly (as recomended in the Kernel Developers Documentation)
Firstly you need to know a few things about the patch programme.
You call it like:
patch -pnum < patchfile.patch
This tells us that the patchfile contains the information to on the files it will be patching, and often there are several. However the folder structure can be different so there is an option to work from up and down a folder tree with -pnum where num is the number of levels to strip off from each filename found in the patch file. This is often -p1 because of the way the patch is created from kernels in separate folders.
There are lots of options but two I find useful are --ignore-whitespace (matches patterns loosely in case tabs and spaces have been munged in the file by email or browsers) and --dry-run (for testing you have the -p value correct)
In this case the patch is in a file at http://linux-kernel.2935.n7.nabble.com/TESTPATCH-v2-xhci-fix-usb2-resume-timing-and-races-td1250796.html . One does not need to be too careful about where to cut out the patch as the patch program is good at only using the important bits but I cut out after the diff -- git line and before the 1.9.1 and put it into a new file xhci.patch - Remember this is NOT the final patch but a test so it is for informatio or experiment on a backed up system! The patch was intended for kernel 4.3 and the second file xhci-ring.c was slightly shorter in 3.19 but the matching in patch found the right place and reported the difference in lines of 9. I did need the --ignore-whitespace and I did a dry run first which looked like the following when it was run from the ubuntu-vivid folder.
patch -p1 --ignore-whitespace --dry-run < xhci.patch
and then removed the --dry-run. As this was my first try I did inspect the files very carefully to see that it looked as if it had worked properly.
Apply a Patch within Git
I now realise it is possible to apply patches within Git and that is arguably the prefered option. To quote ProGit "Using git patch is almost identical to running a patch -p1 command to apply the patch, although it’s more paranoid and accepts fewer fuzzy matches than patch. It also handles file adds, deletes, and renames if they’re described in the git diff format, which patch won’t do. Finally, git apply is an “apply all or abort all” model where either everything is applied or nothing is, whereas patch can partially apply patchfiles, leaving your working directory in a weird state."
git apply --ignore-whitespace --check < xhci.patch
does a trial run as above before you remove the --check. For some reason the --check was not recognised for me so have a new branch is even more important.
Building the kernel
Building the kernel is actually quite easy but is slow - even on my fairly fast i7 laptop it takes nearly an hour, make sure your your working folder is the the root of the kernel source tree and then type the following commands:
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic
I am not sure if the chmod are neccesary but several howto pages include them
fakeroot is an interesting command that wraps some other commands so it looks as if they are being run by root without needing sudo and allows the building of packages.
Changing the version number to avoid overwriting existing unmodified kernels
If the build is successful, a set of at least 4 .deb binary package files will be produced in the folder above the build root folder. These will, by default, have names matching the 'tag' (version number) you have chosen and likewise will be the version number when you install. This means you will be overwriting the existing kernel or risking having an update overwrite it. In order to make your kernel "newer" than the stock Ubuntu Kernel from which you are based you must change the version number.
The version number is extracted from the first line of the debian.master/changelog. To avoid conflicts I incremented the very last part of the number to give a non standard version ie in the example from 3.19.0-32.37 to 3.19.0-32.38 which is not ideal.
The Ubuntu Build Your Own Kernel document instead recommends making your kernel "newer" than the stock Ubuntu kernel from which you are based by adding a local version modifier. Add something like "+helios" or "+xhcifix" to the end of the first version number in the debian.master/changelog file, before building - that is what I will do in the future.
xed debian.master/changelog
This will help identify your kernel when running as it also appears in uname -a. Note also that when a new Ubuntu kernel is released that will be newer than your kernel (which needs regenerating), so care is needed when upgrading. Mint does not automatically upgrade kernels which is an advantage.
Testing the New Kernel
Install the package set on your build machine by something like
sudo dpkg -i linux*3.19.0-32.37.38*.deb
# and reboot
I found that I had 6 files and the ones for tools had dependency problems and are not required so it may be best to only install explicitely the required ones. I may try to find out about the missing dependencies as the linux-tools.... is useful for some diagnostics such as perf.
sudo dpkg -i linux-headers-3.19.0-32_3.19.0-32.37_all.deb linux-headers-3.19.0-32-generic_3.19.0-32.37_amd64.deb linux-image-3.19.0-32-generic_3.19.0-32.37_amd64.deb linux-image-extra-3.19.0-32-generic_3.19.0-32.37_amd64.deb
During rebooting you may need to select the new kernel. If it does not work properly you old kernel should still be available if you have made sure you used a different name or had another stock kernel loaded.
Maintaining your changes, and updating with the latest Ubuntu Kernels
This is where the approach using Git comes into its own as one does not have to download the whole kernel tree every time it is updated. I will not go much further here as this was only intended to be a one off test when I started. To go further we really need to discuss staging, commits, pull and the use of .gitignore
Updating to latest version of git kernels can be done without downloading the whole package again by
git pull
and you will need to commit any changes on your patched branches first. Note you may have conflicts if you have made changes to the files in the kernel. Remember to change debian.master/changelog before rebuilding.
Building Cinnamon, Muffin etc from GitHub (Now slightly out of Date)
My first use of Git was to be able to build Cinnamon on my own machine and test out various development patches.
I spent quite a lot of time reading the PDF version of the Pro Git book, written by Scott Chacon - the link takes you to an online version and there is a download link for the PDF at the left - there is now a second even more comprehensive Second Edition PDF . I put a copy on the phone so I could dip into it and after a number of passes through the early part the more specific instructions started to make sense - I do not like doing things which I do not understand at all. One can also look at http://www.kernel.org/pub/software/scm/git/docs/user-manual.html for more information.
Once I had completed my reading ProGit I switched to the Github help files at https://help.github.com/articles/set-up-git to set up Git on the server. I also set up new email account especially for use with GitHub. I have not bothered with the caching of my GitHub password in Git using a credential helper as it is needed so seldom.
Install and Configuration of Git and some associated programs
I have covered my standard installation and configuration above for kernel patching I will repeat it here so I can break this article up if need be.
sudo apt-get install git gitk leafpad meld
git config --global user.name "your_name_or_your_username_on_github"
git config --global user.email "yourname@your_email"
git config --global core.editor leafpad
git config --global diff.tool meld
git config --global merge.tool meld
git config --global color.ui auto
git config --global push.default simple
The above not only installs Git but its visualiser gitk, a merging program meld and a simple editor leafpad which will only be used by Git.
The first two pieces of configuration are to provide a suitable name and email which is added to every commit you make. If you intend to use GitHub there are advantages to having then the same and using your github username rather than your full name.
The use of a simple text editor called leafpad avoids all sorts of problems in using the same editor for your editing of files and within git. meld is a not only a really good difference tool and is also set to be the default for hand crafting conflicts in a merge. The color.ui auto allows suitable terminal programs to display some information in colour. The push.default simple is to make sure that you only push your master, not branches, to the remote repository by default - it should not be needed with the latest versions of Git but I started before that was the default so some of my early repositories seemed to have a different setting.
Everything in the configuration with --global is available throughout your machine but needs to be redone on other machines. the command git config --list shows your current configuration. Run it outside of a working directory to see just the global settings you have made.
Forking a github repository
Using information at https://help.github.com/articles/fork-a-repo
Click fork when logged in and viewing Cinnamon Repository then
# Clone your fork of the repo into the current directory in terminal
git clone https://github.com/pdcurtis/Cinnamon.git
then move to Muffin Repository and all the others you need and repeat while you are online.
Needed once only however many machines etc you use
Configure remotes
When a repo is cloned, it has a default remote called origin that points to your fork on GitHub, not the original repo it was forked from. To keep track of the original repo, you need to add another remote named upstream:
# Changes the active directory in the prompt to the newly cloned "Cinnamon" directory
cd ~/Cinnamon
# Assign the original repo to a remote called "upstream" which is the standard name
git remote add upstream https://github.com/linuxmint/Cinnamon.git
Needed once only however many machines etc you use
Repeat for muffin and all other repositories that you are going to build it as well.
Keep your local copies up to date
cd ~/Cinnamon
# Fetch any new changes from the original repository, without modifying your files
git fetch upstream
# Merge any changes fetched into your working files
git merge upstream/master
# or do the fetch and merge together from upstream/master (the default)
git pull
I was interested to find the size of the Cinnamon directory
du -ch | grep total
Cinnamon was much smaller than I expected at about 28 Mbytes whilst muffin was 50 Mbytes. Interestingly du gave a much higher figure than the Properties of the folder and these figures rise after compiling to 50 and 70 Mbytes. I suggest allowing 500 Mbytes total as they will grow with the Git repositories.
[ ]
Building your version of Cinnamon
This is now getting to the interesting part where we can do a test to see if we can compile from our own local copy, we can then start to make our own changes or pull others test changes before they are accepted into the Master branch and see the effects. The instructions were found at https://github.com/linuxmint/Cinnamon/wiki/Building and http://community.linuxmint.com/tutorial/view/955 which has some additional comments. This has all been replaced by http://developer.linuxmint.com/reference/git/cinnamon-tutorials/building.html as Cinnamon is now split into many more packages than when I did the following. At that time I started by only building Cinnamon as I am using the nightly updates so I was unlikely to get seriously out of step and only built muffin to check all the proceedures work.
A one-off set-up of repositories and dependencies before building your version of Cinnamon
The next stage is to make sure all the repositories required to compile cinnamon (and muffin) are available. They are usually present in Ubuntu but you can check by looking at the following file /etc/apt/sources.list by
sudo gedit /etc/apt/sources.list
and making sure that the deb-source files are present as well as the others. If you are not using Mint you should ignore the first lines. This is for Ubuntu 12.04 precise and Mint Maya maya and precise will change with to nadia and quantal in 12.10 and so on.
deb http://packages.linuxmint.com maya main upstream import
deb-src http://packages.linuxmint.com maya main upstream import
deb http://archive.ubuntu.com/ubuntu/ precise main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ precise main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ precise-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ precise-updates main restricted universe multiverse
deb http://extras.ubuntu.com/ubuntu precise main
deb-src http://extras.ubuntu.com/ubuntu precise main
Note: These may be in different places in Mint 18 based on Ubuntu 15.04 Xenial as the sources.list has been split up and is in sub-folders
You now need to find out all the required dependencies for cinnamon and muffin and install them by:
sudo apt-get install dpkg-dev
sudo apt-get build-dep cinnamon
sudo apt-get build-dep muffin <other repositories>
sudo apt-get build-dep <next_repository>
Note: dpkg may be installed already. The dependencies needs a 30 Mbyte download and occupies an additional 125 Mbytes for Cinnamon and Muffin together.
Compile and install Muffin and other packages
I have only once carried out these stages as I am using the muffin nightly but if you are working from scratch you will need to check the instructions in one of the links above noting you need to use sudo with dpkg -i for all the packages created. It is possible the order matters if you do not have a version of muffin installed already. The version will change but the following should work if you change the version numbers to match.
cd ~/muffin
dpkg-buildpackage
cd~/
sudo dpkg -i muffin-common_1.1.2_all.deb libmuffin0_1.1.2_i386.deb muffin_1.1.2_i386.deb gir1.2-muffin-3.0_1.1.2_i386.deb libmuffin-dev_1.1.2_i386.deb
Compile and Install the new Cinnamon
NOTE: You will need a version of muffin before compiling Cinnamon. Installing the nightly from the PPA at the very start seems the most sensible way to make sure you have no unresolved dependencies from doing things in the wrong order even if you then overwrite it by installing custom versions of muffin compiled as above.
We are now ready to Compile the new Cinnamon by:
cd ~/Cinnamon
./autogen.sh
dpkg-buildpackage
This is a slow activity but it eventually produces a cinnamon deb file in the parent directory, which you can install with gdebi or dpkg -i.
I had to use dpkg -i as I had the Nightly installed which showed up as a more recent version and gdebi did not allow be to overwrite a newer version so
cd ~/
sudo dpkg -i ~/cinnamon_1.6.7_i386.deb
Note: Version number may need to be updated - check in home folder.
Make sure all your programs are closed as you will need to immediately logout and back in again with the new version - use Alt Ctrl Backspace to force a logout if required
I then found I had the latest version of Cinnamon - without any changes by me of course. This was real progress.
NOTE: You will now need to follow the more complex proceedure at http://developer.linuxmint.com/reference/git/cinnamon-tutorials/building.html for this section because there are more packages involved and the order is critical. Even that only applies to Cinnamon 2.6!
Merging other peoples Pull requests.
One of the main reasons for installing Git and Cinnamon/muffin on my machine is so I can try out the various solutions which are being put forwards for testing. Most use the pull request mechanism even when they are still work in progress so they never get 'pulled' into Upstream for me to access. I tryed this out on a suitable Pull request by running the code provided - in this case:
git pull https://github.com/autarkper/Cinnamon alttab-multi-ws
In this case it turned out to contain 4 commits and had some problems so I wanted to get rid of it. At this point I realised that I should have created a branch for testing it first and it was now in my master branch and very difficult to get shot of. This is because the writers of Git are almost paranoid about protecting one from ever losing any information which is what I wanted to do. In the end I found http://nakkaya.com/2009/09/24/git-delete-last-commit/ and then http://git-scm.com/2011/07/11/reset.html which is by the writer of Git Pro which is my current reference book (PDF) and used.
git reset --hard HEAD~4
HEAD~nn is a shorthand for commit nn before the HEAD. Alternatively you can refer to the SHA-1 of the hash you want to reset to. --soft option will delete the commit but it will leave all your changed files as "Changes to be committed" in the staging area , as git status would put it. I wanted to get rid of any changes to tracked files in the working tree since before the commits so use --hard instead. This is one of the few commands where you can actually lose work you have done so only use it with great care.
More about the use of Git
The following are a few topics to help make this article self contained - there is far more in An Introduction to Git and GitHub with particular application to Cinnamon Applet Development which is really becoming part one of this article.
Making a commit
You can commit (save a snapshot) of all the tracked files in your Working Area with a commit message by:
git commit -a -m "Commit Message"
The -a effectively adds every changed file to the commit and the -m allows the message to be given in the same command.
Switching between branches (git checkout )
When you change to a different branch (or location on a branch) the current contents of the working folder (~/Cinnamon or ~/muffin in this case) reflects the changes which have been made and committed at that point (plus changes made but not committed??) . The branch you have changed to ” is “the branch that will advance when one performs a commit”. The location of the snapshot is referred to as the HEAD.
Create a new branch, for example, when you need to do some testing by
git branch newtestbranch
This creates a new branch from where the HEAD is pointing.
You change branch by:
git checkout existingbranchname
You return to the master branch by:
git checkout master
gitk - a graphic viewer for Git
This program helps enormously in helping one see what one is doing.
sudo apt-get install gitk
Run it in a terminal from within the folder containing the Git repository.
cd ~/Cinnamon
gitk &
The & at the end means it runs as a separate process and the terminal is still available
The following is a typical output after my second attempt at merging in a pull request having created a new branch for my testing (development) and a extra branch for a particular test when I was investigating a menu problem with Cinnamon (menu-issue). At this time there have been no additional changes in 'upstream' to fetch and merge.
If this is all becoming a bit of a mystery then have a look at http://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/ which goes through all the steps on how to create a branch to do some work, after it has been test merge it into the branch you are using for long term development (development) , delete the testing branch and how to push the extra work (commits) back to your branch on the remote server (origin) and takes one through what gitk shows after every step in a simple example.
Reference Information on Git and use of GitHub
Git is the distributed version control system which was written by Linus Torvalds initially for version control of the Linux Kernel. It is the most widely used version control system in use today and is used by Mint and Cinnamon.
There is a lot more information on Git and its use in my document An Introduction to Git and GitHub with particular application to Cinnamon Applet Development which is really becoming part one of this article. This not only supports users of my applets but also those wishing to set up a version control system using Git and GitHub for their own software development and it also serves as a part 1 for this document.
Other links and documents I have found useful are:
- Wikipedia - GIT Software - a good introduction to Git history
- The Pro Git book, written by Scott Chacon available online and in a Second Edition as a PDF . This is the best of the references tin my opinion but be warned it takes about 150 pages to cover the most important areas (Chapters 1 - 3 and the start of Chapter 6)
- Git Magic - A more informal but usefull approach - you may need to google for alternative downloads - last checked 7 Aug 2016
- Git from the Bottom Up - A good cross check on your understanding of fundamentaals but not the place to start.
- Github guide - . Github.com is the server used for all the cinnamon system software which is free for open source software.
- Linux Kernel Team User Manual - from the Linux Kernel Team
- Use gitk to understand Git - a very useful tutorial on use of gitk part 1
- Use gitk to understand Git merge and rebase useful tutorial for an essential tool part 2
- Differences between Git Fetch and Git Pull- A Stack Overflow question with some useful information and graph
- GitHub Git Cheat Sheet - Two sides of A4 to print gives you 40 commonly used command line instructions for reference including a one line summary.
- Tower's Git Cheat Sheet - with care you can print this on one page and it also has some 'best practice' notes. Does not completely overlap the above sheet so both are useful.
Before You Leave
I would be very pleased if visitors could spare a little time to give me some feedback - it is the only way I know who has visited, if it is useful and how I should develop it's content and the techniques used. I would be delighted if you could send comments or just let us know you have visited by Sending a quick Message to me
