Wednesday, June 24, 2009

Conclusion: Restore from USB to minimal EeePC Linux system

(cont'd from: Part V: Yet another test with 1.6 French iso)

My Last Tests with 900 and 1000H iso

In the past 2 weeks, I have been testing out the different scenarios, including with the 1000H iso and 900 iso. Now, all known bugs have been fixed and all conditions tested and verified working.

I have also broken down the "extra.sh" script into separate scripts (extra*.sh) for each extra job. This would allow the user to easily delete scripts he don't need and add new scripts. Just like the "remove*.list" files, the main script will run the extra scripts one by one after removing programs according to the "remove*.list" you specified, and before resizing.

The 900 iso testing went very smooth, but the testing with the 1000H iso is particularly troublesome. I ran into all kinds of different errors, e.g. udev not working (cannot detect USB drives), asus-launcher not launching the Easy Mode desktop, power button not working ... Only after many many trials, I tracked this down to the extra scripts I have written trying to deactivate Asus update and install wireless and LAN drivers for my 4G as per Howto: Fix network problem using 1000H iso on a 4G. I didn't solve this problem - very strange. So in the end, I removed these 2 extra scripts. [EDIT 2009/6/29: I have finally worked this out and put these extra scripts back into my script package. Just a silly mistake: forget to give the scripts exec permission after editing them.]

I have also tested with different extra% and found that with 2% or 3% extra in one case and another, I have achieved 100% disk usage in sda1 after shrinking. With 1%, resize2fs (and e2fsck) will report failure. So I think the original 5% extra should be a safe and economical choice which will result in around 97% used.

So if you have an EeePC other than the 701 series, even if you are just uninstalling 1 or 2 packages, with this script, you will be able to squeeze some extra disk space (up to 300~500 MB depending on your model) because many models start with something like 87% (even 76%) used in sda1. [NB. If you don't uninstall anything, the script will NOT shrink the partition sda1.]

The time to do this ranges from 17 mins with a 701 4G iso to 25 mins with a 1000H iso, provided that you have a fast USB device. With a slow one, it may take longer to dump the Linux recovery image from the USB device which normally only takes 5 minutes.

Summary of Errors Encountered

1) Couldn't umount after removing staroffice from the 1000H iso (SOLVED)
Umount: Couldn't umount /mnt: Inappropriate ioctl for device
I don't quite understand this error which has troubled me for a few days. After carefully tracking it down, I found that this comes after uninstalling staroffice* (regardless of removal of other programs and has nothing to do with total file size removed).

Finally, I have been able to sort this out with a lazy umount (umount -l) and wait for 15 seconds immediately after uninstalling staroffice. So I have to add a series of umount and mount commands after each apt-get operation of package removal.

2) Couldn't move some symbolic links in sda1 (SOLVED)
Moving /usr/lib/[b-l] ...
mv: unable to rename '/mnt/usr/lib/libevent.so.1': No such file or directory
mv: unable to rename '/mnt/usr/lib/libgcj_bc.so.1': No such file or directory
mv: unable to rename '/mnt/usr/lib/liblber.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libldap.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libldap_r.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libvolume_id.so': No such file or directory
I had these errors moving symbolic links since the beginning. Although they do not seem to cause any problem, to play safe, I want to eliminate them. So I log the error messages and process them to extract the link name and link target filename, then delete the original symbolic links and re-create them.

I have also logged the "ls -l" output for all these links before and after the operation so that the user can check and verify afterwards. [NB. The timestamp for these files will become different after this operation.]

3) With the 701 4G iso, after restoring to minimal system, there is a hissing sound from the speaker as reported many times by others in EeeUser forum (both Linux and winxp). This will not affect music playing, and will go off if you mute the speakers. Since this occur to both Linux and winxp, I am inclined to think it is hardware related (perhaps BIOS?). I have not looked into this yet.

Overview of Script Operation

As an overview, here is what the script will do:
1) Delete all partitions in sda1. Then create one primary partition, and format it. [Note: This is for to correct any errors in the partition table and filesystem. The original script from Asus will fail in such cases.]
2) Dump the Linux system image to the 1st SSD (MBR and sda1).
3) Create the user partition (sda2) and other partitions, format and label them.
4) For those models with a 2nd SSD, create, format and label the HOME partition (sdb1).
5) Uninstall programs from the system partition (sda1) as per the remove*.list files.
6) Run extra jobs as per the extra*.sh scripts.
7) Defragment the filesystem in sda1 by moving files to sda2 and back.
8) Calculate available and used disk space, and shrink sda1, allowing 5% extra.
9) Reboot.

Steps of Procedure

1) Prepare an Asus EeePC USB Recovery Disk. Read your EeePC User Manual or my other blog: Create an Asus EeePC System Recovery USB disk.
[NB. You will need a 1 GB USB disk for the EeePC System Recovery USB disk working with the 701 and 900 iso, but 2 GB for the 1000H and version 1.6 iso.]


2) Download my script package restore-minimal-701-4G.tgz or restore-minimal-1.6-701-fr.tgz and extract the files to your System Recovery USB disk. [NB. Some people have reported problem extracting the files in Microsoft Windows with winzip and pkunzip.]

3) Edit the "remove*.list" files in the folder "removes/" according to the factory installed package list in your EeePC Linux version. You can run the command "dpkg -l" in a terminal to see what packages are currently installed. Be sure not to uninstall any system package or else you won't be able to boot.
[NB. Do not use Windows Notepad to edit the files. Use Notepad2 instead if you want to do it in Windows. The files should be saved with UNIX style "End of Line" instead of MSDOS text format.]

If you are using an EeePC Linux version I have tested, you can download and start from my "remove list" files:




















ModelVersionLanguagesRemove File List
701 4G1.0.1en tw cnremove_files-701_4G.tgz
900 1.1.0.66en tw cn
remove_files-900.tgz
10001.6.0.104
en
remove_files-1000H.tgz
7001.6.1.5-a4enremove_files-701_en_US-1.6.1.5-a4.tgz
700 1.6.1.39fr en
remove_files-701_1.6_fr.tgz

4) Delete the extra*.sh scripts you don't need. Please note that there are different sets of extra*.sh scripts in restore-minimal-701-4G.tgz and restore-minimal-1.6-701-fr.tgz. Some extra scripts in the latter are only for 1.6 version, e.g. disable auto update. There are description of what each extra script does in the SUMMARY file in the extra/ folder.

5) Boot up from the USB Recovery disk - Press [ESC] at boot up and select the USB disk as 1st boot device, and let the script run.

You will be prompted during the process by apt-get to confirm removal of programs, and also for dumping the Linux system recovery image and shrinking sda1.

For details, please refer to my previous tests on this:
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso

Screenshots

1) Booting up and dumping the Linux recovery system image to sda1

2) Removing programs as per remove*.list

3) Defragment sda1 by moving files from sda1 to sda2 and back

4) Fixing mv problem with symbolic links

5) Final disk space in system and user partitions



What's Next?

While working on this restoring to minimal system, I have been thinking with the same method, it should also be possible to uninstall applications from the read only partition of the union filesystem (i.e. sda1).

The difference between the two approaches will be:
  1. No need to dump the Linux recovery image to sda1.
  2. Back up the user partition (i.e. sda2), if necessary. [NB. Since the read-only system partition sda1 can be easily restored from the Linux recovery image, I think there is no need to back it up.]
  3. Instead of uninstalling programs in the busybox chroot environment booting from USB, we should boot into a read-write union filesystem to implement the changes, so that dpkg and apt-get running under the union filesystem will be fully aware of the changes. [NB. In a read-write union filesystem, if you upgrade a package, old files in sda1 will be deleted and new files will always be written to sda2.]
  4. Need to back up all files in the user partition (i.e. sda2) to the USB device before shrinking sda1, and restoring them after operation completed.
So, it should not be possible to do this automatically with a script as in this restore to minimal approach. Instead, I think I need to split the main script into separate scripts including:
The user will need to boot into the read-write union filesystem and run synaptic to uninstall packages he don't need. And if he want to retain the F9 restore function, better not to upgrade packages that will upgrade system libraries, only uninstall programs he don't need and/or upgrade applications like openoffice that is not loaded at system boot up.

Also, the user will need to make sure there is enough free disk space in sda2 for upgrading some big packages, e.g. openoffice, because all new files will be written to sda2 before you can shrink sda1 to reclaim the free space from the old files deleted from sda1.

This also applies to force re-installing factory installed packages. The original files will be deleted from sda1 and write to sda2, so effectively moved from the read-only sda1 to the read-write sda2. So you can do whatever you like with it later.

This will be my next project.

Part I : My 1st Trial
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system

Thursday, June 18, 2009

Yet another test: Restore to minimal with 1.6 French iso

(cont'd from Part IV: 2nd Test restoring to minimal with 1.6.1 701 en iso)

One more test again tonight, with the 1.6 French iso. The approach is basically the same. Only difference is I need to modify the "extra.sh" script to automatically change the line "DefaultLanguage=fr_FR" to "DefaultLanguage=en_US", and also edit /etc/apt/sources. list to change the line with "fr-channel" to "en". Here is the complete script package for this iso: restore-minimal-1.6-701-fr.tgz.

Apparently, this version has less factory installed packages, thus resulting in less to-be-removed packages - only a total of 676MB.
EeePC Linux 1.6.1.39 701 fr
Version: Eee PC 1.6.1.39
Build Info: 2008-07-08 05:45

remove_acroread.list 127 MB Acrobat reader
remove_drivers.list 537 kB Additional hardware drivers
remove_fr.list 59.7 MB French locales
remove_games.list 7094 kB Programs in GAMES folder of PLAY tab
remove_internet.list 119 MB Programs in INTERNET tab
remove_kids.list 59.2 MB Programs in LEARN tab
remove_ooo.list 235 MB Openoffice.org
remove_play.list 52.6 MB Programs in PLAY tab
remove_settings.list 16.1 MB Selected programs in SETTINGS tab

Again original partition layout only used 87% in sda1 (now 97%). After resizing, sda1 reduced from 300 to 156 cylinders, and sda2 increased from 184 to 328 cylinders (original 1.4 GB to 2.6 GB now).

"fdisk -l" output:
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1 1 156 1253038+ 83 Linux
/dev/sda2 157 484 2634660 83 Linux

Re-installing Removed Packages from downloaded repository

As for re-installing removed packages, experience is basically the same as with the 701 en-US 1.6.1.5-a4 iso in my previous test. The only difference is this time I am re-installing from a local repository of Asus Update 1.6 repo tree I have downloaded. Please refer to my previous blog for details.

One point worth mentioning is that do not update apt cache before you edit and change /etc/apt/sources.list and replace the original http://update.eeepc.asus.com entries with the local repos entries.
deb file:/media/"D:"/1.6/ common main
deb file:/media/"D:"/1.6/ p701 main
deb file:/media/"D:"/1.6/ en main
Otherwise, synaptic will keep asking to download packages from http://update.eeepc.asus.com and also remember firefox3 as the highest available version for firefox.

The packages of trouble are the same: mplayer that needs to be download and installed manually. The rest is very easy. In fact, this time I just re-install the packages according to the remove lists. Very easy!
sudo apt-get install `cat remove_drivers.list`
sudo apt-get install `cat remove_games.list.list`
sudo apt-get install `cat remove_games.list`
sudo apt-get install `cat remove_kids.list`
sudo apt-get install `cat remove_ooo.list`
sudo apt-get install `cat remove_settings.list`
For "remove_play.list", I need to download and install mplayer first.
wget http://update.eeepc.asus.com/p701/pool/mplayer_1.0%7Erc1-12etchxandros10_i386.deb
sudo dpkg -i /home/user/mplayer_1.0~rc1-12etchxandros10_i386.deb
apt-get install `cat remove_play.list`
And for packages in "remove_internet.list", I don't know why, but I have to install libpurple0 and pidgin in this sequence.
cat remove_internet.list
apt-get install skype thunderbird asus-usbhandset firefox adobe-flash-player
apt-get install libpurple0
apt-get install pidgin
So like this, re-installation is much faster than with previous tests. And I ended up with 1319MB free disk space.

Like this, it should be possible to also automate the re-installation process with a script!

Part I: My 1st Trial
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system

Monday, June 15, 2009

Build a local repository of EeePC Asus Update

With the success of my approach to restoring EeePC Linux from USB to a minimal system, in the coming few days, I am expecting to run more tests on different iso versions I have on hand, and probably removing and re-installing different sets of packages.

To speed up the re-installation process, I think it is better to download all the packages for the EeePC Linux version 1.6 I am working on from http://update.eeepc.asus.com/ and build a local repository, so that I don't need to spend long time to download the packages for re-installation, and may be even automate the process with a script.

Download Asus Update packages with wget

In Linux, there is a handy tool called wget that comes pre-installed in every distribution. There are also GUI frontends for wget (e.g. Gwget from the Debian repositories), but I think it is simple enough just to run it in the terminal. If you prefer to work in Microsoft Windows, there is also a Windows version wGetGUI.

Take the example of a 701 en_US 1.6 version, first you need to know the URL where the packages are hosted, and this information can be obtained from your /etc/apt/sources.list, e.g.
deb http://update.eeepc.asus.com/1.6 common main
deb http://update.eeepc.asus.com/1.6 p701 main
deb http://update.eeepc.asus.com/1.6 en main
This means that the *.deb packages you need will be hosted in URLs:
http://update.eeepc.asus.com/1.6/pool/common
http://update.eeepc.asus.com/1.6/pool/en
http://update.eeepc.asus.com/1.6/pool/p701
And with this command, wget will download all files from the URL you specified:
wget -r -w 20 -nd -np -l 1 http://update.eeepc.asus.com/1.6/pool/common
Here, the option "-w 20" ask wget to wait 20 seconds before retrievals. If you are not in a hurry, this is considered a good practice so that you don't stress the server too much occupying all its bandwidth. Be considerate to others!

Also, the option "-l 1" set the depth level to 1, so that wget will not get packages from "Parent Directory". But you will have to run this command for every folder in the URL lists you want to download. e.g. There is a sub-folder "openssl" in "common", and you will have to run wget again to download files in this folder:
/media/D:/1.6/pool/common/openssl> wget -r -nd -np -l 1 http://update.eeepc.asus.com/1.6/pool/common/openssl
--21:37:02-- http://update.eeepc.asus.com/1.6/pool/common/openssl
=> `openssl'
Resolving update.eeepc.asus.com... 211.72.249.195
Connecting to update.eeepc.asus.com|211.72.249.195|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://update.eeepc.asus.com/1.6/pool/common/openssl/ [following]
--21:37:02-- http://update.eeepc.asus.com/1.6/pool/common/openssl/
=> `index.html'
Reusing existing connection to update.eeepc.asus.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 1,279 (1.2K) [text/html]

100%[====================================================>] 1,279 --.--K/s

21:37:03 (40.05 MB/s) - `index.html' saved [1279/1279]

...

--21:37:08-- http://update.eeepc.asus.com/1.6/pool/common/openssl/openssl_0.9.8c-4etch5_i386.deb
=> `openssl_0.9.8c-4etch5_i386.deb'
Reusing existing connection to update.eeepc.asus.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 1,015,578 (992K) [application/x-debian-package]

100%[====================================================>] 1,015,578 617.73K/s

21:37:10 (615.75 KB/s) - `openssl_0.9.8c-4etch5_i386.deb' saved [1015578/1015578]

FINISHED --21:37:10--
Downloaded: 3,991,869 bytes in 4 files
/media/D:/1.6/pool/common/openssl>
For more information on wget options, read the GNU Wget Manual.

What I did is I put the commands into a script and ran it before I left for work. When I returned home, the download has already finished. A total of 2.96 GB has been downloaded.

Build a local Repository on an USB stick for my EeePC

For the local repository, I need an USB stick of at least 4 GB, where I can copy the 3 folders "common", "en" and "p701", and the deb packages they contain. You can place it in any folder, but my choice is "1.6/pool".

Here, most of the packages are in the folder "common", with some packages in the "p701" and just a few in the "en" folders. "p701" folder is for packages specific to 701 model, and "en" folder is for packages specific to US English version.

I have deleted the sub-folders "firefox3", "p901-system-update" and "staroffice", and also search and delete deb packages for locales other than "zh-tw".

The folder "firefox3" contains the "firefox-upgrade" package which will installed firefox3, flashplayer10, upgrade libc6 to version 2.7-13, and gtk libraries to 2.12.1. This is convenient for those who like firefox3, but will cause problem when I install packages from Debian etch repositories later, e.g. vlc. So I prefer alternative solutions.

To build the local repositories, you can refer to this APT HOWTO: How to use APT locally.

Basically, what you need to do is: first create the folder tree for saving the package list to be generated, then run the command "dpkg-scanpackages" which will read the control files in each deb package, e.g. "pool/common/*.deb", and generate the file "Packages":
mkdir -p dists/common/main/binary-i386
mkdir -p dists/en/main/binary-i386
mkdir -p dists/p701/main/binary-i386
dpkg-scanpackages -m pool/common /dev/null > dists/common/main/binary-i386/Packages
dpkg-scanpackages -m pool/en /dev/null > dists/en/main/binary-i386/Packages
dpkg-scanpackages -m pool/p701 /dev/null > dists/p701/main/binary-i386/Packages
Here is an example of the terminal output running this:
/media/D:/1.6> dpkg-scanpackages -m pool/en /dev/null > dists/en/Packages
** Packages in archive but missing from override file: **
kaffeine-dtv v4ldvbpb voice-command-en

Wrote 3 entries to output Packages file.
/media/D:/1.6>

To use this local repositories, you will need to edit /etc/apt/sources.list and add the following entries:
deb file:/media/"D:"/1.6/ common main
deb file:/media/"D:"/1.6/ p701 main
deb file:/media/"D:"/1.6/ en main
Then update apt cache:
/media/D:/1.6> sudo apt-get update
Ign file: common Release.gpg
Ign file: p701 Release.gpg
Ign file: en Release.gpg
Ign file: common Release
Ign file: p701 Release
Ign file: en Release
Ign file: common/main Packages
Ign file: p701/main Packages
Ign file: en/main Packages
Reading package lists... Done
/media/D:/1.6>
If no error comes up, you can now fire up synaptic and check if the local repositories are available. Now, you will be able to re-install all the packages you have removed in a swift!

Saturday, June 13, 2009

Part IV: 2nd Test restoring to minimal EeePC Linux with 1.6.1 701 en iso

(cont'd from Part III: Re-installing Removed Packages ...)

My next test that follows is to build a minimal EeePC Linux system on my 4G with a 701 en-US 1.6.1.5-a4 iso.

Prepare the System Recovery USB Disk

With the downloaded iso, first I burn it into a DVD, then run BootTool to create the System Recovery USB disk. But the operation was not successful, "P701L.gz" and "user_start.dat" cannot be copied to the 1GB USB disk.

This is the first time I experience this error though I have heard about it for long time already. I tried it twice not successful, not even by manually copying the files from the DVD to the USB stick. So I ended up using another USB disk (the 2GB one I use to use for such testing).

Next I download the script package restore-minimal-701-4G.tgz I have uploaded to my website, and extract it to the USB disk with 7-zip File Manager.

Edit the Remove Package Lists

The next thing to do is to edit the remove*.list files for the packages I want to remove. But since I have no idea what packages are there in this EeePC Linux version, I first restore from USB with the original bootscript (select the 1st entry at the grub boot menu), then boot it up to see what is in it.

With the "dpkg -l" or "dpkg-query -W" command, I check what packages are installed and edit the remove*.list files accordingly. [Note: I have uploaded the dpkg -l list to this Wiki.] Basically, unlike in the previous test, there are no Chinese locale packages in this version, so no more "remove_zh_tw.list" and "remove_zh_cn.list". Also scim has been replaced by gcin. The resulted remove lists are slightly different to the 701 4G version. There are less packages to remove - only a total of 683 MB.
/media/D:/removes> ls
README remove_games.list remove_kids.list remove_settings.list
remove_acroread.list remove_gcin.list remove_ooo.list SUMMARY
remove_drivers.list remove_internet.list remove_play.list
I also take this opportunity to test the remove list one by one uninstalling them with apt-get to see if any error will occur during the process. [NB. Since I still have union filesystem, nothing will really get uninstalled this time.]
/media/D:/removes> sudo apt-get remove `cat remove_games.list`
Reading package lists... Done
Building dependency tree... Done
Note, selecting xandros-frozen-bubble-helper for regex 'frozen-bubble*'
Note, selecting frozen-bubble-lib for regex 'frozen-bubble*'
Note, selecting frozen-bubble-data for regex 'frozen-bubble*'
Note, selecting frozen-bubble for regex 'frozen-bubble*'
Note, selecting planetpenguin-racer-extras for regex 'planetpenguin-racer*'
Note, selecting planetpenguin-racer for regex 'planetpenguin-racer*'
Note, selecting planetpenguin-racer-data for regex 'planetpenguin-racer*'
The following packages will be REMOVED:
crack-attack fb-music-high frozen-bubble frozen-bubble-data gnome-sudoku kdegames-card-data
kpat ktuberling libkdegames1 ltris planetpenguin-racer planetpenguin-racer-data
xandros-frozen-bubble-helper xandros-gnome-sudoku-helper
0 upgraded, 0 newly installed, 14 to remove and 12 not upgraded.
Need to get 0B of archives.
After unpacking 59.3MB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 64338 files and directories currently installed.)
Removing crack-attack ...
Removing xandros-frozen-bubble-helper ...
Removing frozen-bubble ...
Removing fb-music-high ...
Removing frozen-bubble-data ...
Removing gnome-sudoku ...
Removing kpat ...
Removing kdegames-card-data ...
Removing ktuberling ...
Removing libkdegames1 ...
Removing ltris ...
Removing planetpenguin-racer ...
Removing planetpenguin-racer-data ...
Removing xandros-gnome-sudoku-helper ...
/media/D:/removes>
For those who are interested, here is the set of remove file lists I have ended up with: remove_files-701_en_US-1.6.1.5-a4.tgz.

Before and After

Again, I have compared the the "df" and "fdisk -l" outputs before and after, showing the free disk space and the partition layout before and after the process:

1) Original 13% (!!?) free disk space in sda1, total filesystem size of 2371235k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda1 2371235 2059136 312099 87% /mnt
2) Now 3% free disk space in sda1, Used = 1402473k, i.e. freed up 656 MB. Also filesystem size reduced to 1446438k, i.e. sda1 40% smaller
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda1 1446438 1402473 43965 97% /mnt
3) Original disk partition layout: Exactly the same as 701 4G, sda1 occupies 300 cylinders and sda2 occupies 184 cylinders. I guess Asus has changed the packages but have not resized the partition accordingly, so resulting in so much not usable free disk space in sda1.
Disk /dev/sda: 4001 MB, 4001292288 bytes
255 heads, 63 sectors/track, 486 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 300 2409718+ 83 Linux
/dev/sda2 301 484 1477980 83 Linux
/dev/sda3 485 485 8032+ c W95 FAT32 (LBA)
/dev/sda4 486 486 8032+ ef EFI (FAT-12/16/32)
4) After resizing: read-only sda1 only occupies 183 cylinders and read-write sda2 enlarged to 301 cylinders
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1 1 183 1469916 83 Linux
/dev/sda2 184 484 2417782+ 83 Linux
So much more usable disk space for you.

Moreover, with this hack you are free to delete and re-install packages as you like. Not like with union filesystem when you upgrade a package, the old files are still eating up your precious disk space in the read only partition. This is especially true if you plan to upgrade openoffice, Acrobat Reader, etc.

Re-installing Removed Packages

For the purpose of testing, I have re-installed all removed packages again.

Re-installing removed packages is more complicated for this version. I tend to believe that this is an alpha version, because most of the factory installed package version are not found in the 1.6 en respositories tree on Asus Update website. Instead, they scatter everywhere in the Asus Update website - some from 900, some from 1000H, ... everywhere! I have to use the Advanced search of Google to locate them within http://update.eeepc.asus.com/.

Also, the original respositories tree is for EeePC Linux version 1.6.4 p701 en, but there are not many packages in that repos tree. Probably just an experimental version. If I just run an "sudo apt-get update" and run the synaptic package manager, there are hardly any packages available. So I have to edit /etc/apt/sources.list and add the 1.6 repos entries:
deb http://update.eeepc.asus.com/1.6 common main
deb http://update.eeepc.asus.com/1.6 p701 main
deb http://update.eeepc.asus.com/1.6 en main
Then when I fire up synaptic and re-install the packages running through the list in the "removed.log", it turned out that most packages can be installed easily - easier than with 701 4G iso.

Some tips to do it faster:
- Games can be ticked from the [Section] View > [Games and Amusement] of synaptic
- Same for those educational programs - in [Science] and [Mathematics] section


Also, if you plan to do it over and over, again and again, you may want to save or backup the *.deb in the folder /var/cache/apt/archives/ to somewhere else. Installing a program won't take much time, it take much longer to download it.

EDIT 2009/6/17: See my blog of Jun 15, 09 for how I build a local repository of Asus Update to speed up this re-installation process.


Packages of trouble

1) Installing smplayer popup with this error message because mplayer is not available in any of the 1.6 repo trees!

I have to manually download it from the 701 pool: mplayer_1.0~rc1-12etchxandros10_i386.deb
and install it in File Manager:

This installation will end up with an error message because mplayer depends on mplayer-skin but mplayer-skin has not yet been installed (because it also depends on mplayer!). When you run synaptic later, it will complain there is a broken package.

Now just install mplayer-skins, and the dependency will be resolved.

2) Firefox - need to force version to 2.0.x because 3.0.4 is also available on the 1.6 repo tree but requires libc6 and a lot of libraries upgrade and can only be done by running the firefox-upgrade meta package.

To force version, first highlight the package you want to install, then select from the pull down menu [Package] > [Force Version],

and select the 2.0.x versions.


3) adobe-flash-player - the infamous problem with flashplayer10 on EeePC Linux. Again we need to force version to 9.x instead of the higher 10.x versions. If you want the 10.x version, you have to install it as part of firefox-upgrade (to version 3.0.x) because it requires a higher libc6 version than the default one.


4) gcin (Chinese input Method) - This is the most troublesome. All versions available from the 1.6 repo trees require a libc6 upgrade which is of course undesirable (though possible). So I downloaded it from the p701 repos tree: gcin_1.3.4-2asus4_i386.deb and gcin-qt3-immodule_1.3.4-2asus4_i386.deb.

To install them, open a terminal and enter this command:
deb/home/user> ls
gcin_1.3.4-2asus4_i386.deb gcin-qt3-immodule_1.3.4-2asus4_i386.deb Trash/ My Documents/
/home/user> sudo dpkg -i *.deb
Selecting previously deselected package gcin.
(Reading database ... 71993 files and directories currently installed.)
Unpacking gcin (from gcin_1.3.4-2asus4_i386.deb) ...
Selecting previously deselected package gcin-qt3-immodule.
Unpacking gcin-qt3-immodule (from gcin-qt3-immodule_1.3.4-2asus4_i386.deb) ...
Setting up gcin (1.3.4-2asus4) ...

Setting up gcin-qt3-immodule (1.3.4-2asus4) ...
/home/user>
The installation went alright, but gcin is still not showing up in the taskbar. So I open synaptic again and install "xandros-gcin". And on re-starting X by pressing [Ctrl]-[Alt]-[BkSp], now I can input Chinese.

Final Comments

After these testings, I can now confirm that this restore to minimal script is working, though not perfect. There seems to be still some minor bugs, but basically it works. And since it take quite some time to run it once, I don't think I can run many test on this.

The % of free disk space in sda1 I ended up with varies with the different iso and what packages you want to remove, and is rather out of my control. With the 701 4G, I achieved 92%, and this one 97%. Compared with the disk space reclaimed shrinking down sda1, I think these figures are acceptable.

If I have time to further test on other iso, I will further post my remove file lists. But I think some Linux expert may be able to come up with a more comprehensive remove file list, knowing what libraries can be safely removed.

And so this concludes this project of mine.

P.S. With the 701 en-US 1.6.1.5-a4 iso, I do not end up with the hissing sound in the minimal system.

Part I: My 1st Trial
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system

Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G

(cont'd from Part II: Success - Restoring my EeePC from USB to a Minimal System)

After you have restored from USB to a minimal system with my my restore script package restore-minimal-701-4G.tgz, you can re-install packages you want with synaptic package manager.

I have tried to write a script to re-install them all in one stroke but found that it is not so simple because the original factory version of many packages are not available from Asus Update website. Instead, only higher versions can be found. So you are forced to upgrade those packages.

With my 701 4G, except for a few packages, most removed packages can be re-installed easily with the synaptic package manager.

Below is the missing list:
acroread-cn-tw=7.0.9-1xandros3
acroread=7.0.9-1xandros2
hsfmodem-modules-2.6.21.4-eeepc=7.60.00.04oem-8
kpat=4:3.4.2.11-11
kdegames-card-data=4:3.4.2.11-11
libkdegames1=4:3.4.2.11-11
planetpenguin-racer=0.3.1-8xandros3
planetpenguin-racer-data=0.3.1-8xandros3
ktuberling=4:3.4.2.11-11
skype=1.4.0.114-1xandros1
firefox=2.0.0.7-1asus1
firefox-locale-zh-cn=2.0.0.7-1asus1
firefox-locale-zh-tw=2.0.0.7-1asus1
pidgin=1:2.0.2-0ubuntu1+asus8
pidgin-data=1:2.0.2-0ubuntu1+asus8
fbreader-qt4=0.8.4a1xandros8
asus-usbhandset=1.0-17
kalzium=4:3.4.2.12-19
kstars=4:3.4.2.12-19
kstars-data=4:3.4.2.12-19
kmessedwords=4:3.4.2.12-19
khangman=4:3.4.2.12-19
kbruch=4:3.4.2.12-19
kig=4:3.4.2.12-19
kmplot=4:3.4.2.12-19
openoffice.org=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-base=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-calc=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-common=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-core=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-draw=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-help-en-us=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-help-zh-cn=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-help-zh-tw=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-impress=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-java-common=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-kde=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-l10n-zh-cn=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-l10n-zh-tw=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-math=2.0.4.dfsg.2-7etch2eeepc1
openoffice.org-writer=2.0.4.dfsg.2-7etch2eeepc1
smplayer=0.5.21-0xandros23
amarok=1.4.3-3xandros38asus27
amarok-engines=1.4.3-3xandros38asus27
amarok-helix=1.4.3-3xandros38asus27
amarok-xine=1.4.3-3xandros38asus27
gwenview=1.4.2-1xandros7
gwenview-i18n=1.4.2-1xandros5
scim=1.4.4-7asus4
scim-modules-table=0.5.6-2eeepc1
scim-tables-zh=0.5.6-2eeepc1
libscim8c2a=1.4.4-7asus4
xandros-antivirus=2.0.62-2
libclamav2=0.91.2-4
diskutil=1.0.0-0xandros17
asus-userdiag=1.0.16
thunderbird-locale-zh-cn=1:2.0.0.6-1asus1
kde-i18n-zhcn=4:3.4.2.84-2
xandros-i18n-zhcn=1.159-1
thunderbird-locale-zh-tw=1:2.0.0.6-1asus1
kde-i18n-zhtw=4:3.4.2.84-2
xandros-i18n-zhtw=1.159-1


Packages of trouble and how I deal with them

1) frozen-bubble - need to force version to original 2.1.0-1


2) clamav - depend on libclamav2 (>= 0.91.2) but is not available in the 701 repositories tree.

I have to manually download libclamav2_0.91.2-4_i386.deb from Asus Update version 1.6 common repo tree and install with right click > [Install DEB file] in File Manager before I can use synaptic to install clamav and xandros-antivirus.

3) There are a few packages that can not be directly installed, e.g. firefox, openoffice, ... , etc. This is probably because the latest version need extra or different libraries not provided in the factory EeePC Linux version.

To install them, search in synaptic and look for their update pack provided by Asus, e.g. "asus-openoffice-update-1".

NB. If you have a 701 4G, searching in synaptic may result in this error:
E: Unable to parse package file /var/lib/dpkg/status (2)
E: Unable to lock the download directory
Avoid using the [Search] button in synaptic or read this wiki for solution.

Part I: My 1st Trial
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system

Part II: Success - Restoring my EeePC from USB to a Minimal System

(cont'd from Part I: My 1st Trial)

After struggling for two weeks with failures after failures, I have finally worked it out. It turned out that simply by moving the files from sda1 to sda2 and back is enough to defragment the filesystem. But depending on the packages you removed and where the files are originally located, the resulted disk fragmentation may vary and it need some trial and error to determine the end block of sda1.

For the brave ones who would like to join my adventure, you can download my script package restore-minimal-701-4G.tgz and extract the files to your EeePC System Recovery USB disk. [NB. Do not use winzip to extract the files] Then reboot and run it like what I have described in Restore EeePC from Recovery DVD without unionfs or Boot Xandros from USB for Dummies.

Scenario I have Tested

This is only preliminary success - different scenarios are not yet thoroughly tested. So far only tested successful with the following setup:
Hardware: EeePC 4G
Recovery EeePC Linux disk image: 701 4G iso
Set of Uninstalled Programs: Minimal system (as per remove*.list files I have enclosed in my script package)
Allow extra disk space while shrinking filesystem = 5% (achieving final available disk space of 92% vs. original of 96%)
[NB. If you shrink the filesystem too much with resize2fs, it is not possible to redo it. You will have to start all over again. So it is up to you to experiment if you want to maximize use of disk space in the read only partition.]

If you have another version of EeePC Linux other than the EeePC Linux version 1.0 (for 701 4G) I have tested, you will need to check and change the package names in the remove*.list files according to your version. Run "dpkg -l" to list packages installed in your system and compare with my remove list.
[Remember: Do not edit the files with Windows Notepad. Use
Notepad2 instead if you want to do it in Windows. The files should be saved with UNIX style "End of Line" instead of MSDOS text file.]

Before and After

Here are the "df" and "fdisk -l" outputs before and after the process. These show the free disk space and the partition layout before and after the process:

1) Original 5% free disk space in sda1, 2147308k used out of 2371820k total filesystem size
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda1 2371820 2147308 104028 95% /mnt
2) Now 8% free disk space in sda1, Used = 1367640k, i.e. freed up 780 MB. Also filesystem size reduced to 1534320k, i.e. 35% smaller
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda1 1534320 1367640 119932 92% /mnt
3) Original disk partition layout: sda1 occupies 300 cylinders and sda2 occupies 184 cylinders
Disk /dev/sda: 4001 MB, 4001292288 bytes
255 heads, 63 sectors/track, 486 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 300 2409718+ 83 Linux
/dev/sda2 301 484 1477980 83 Linux
/dev/sda3 485 485 8032+ c W95 FAT32 (LBA)
/dev/sda4 486 486 8032+ ef EFI (FAT-12/16/32)
4) After resizing: sda1 only occupies 193 cylinders and sda2 enlarged to 291 cylinders
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1 1 193 1550241 83 Linux
/dev/sda2 194 484 2337457+ 83 Linux
Please note that I have tested it a few times and the numbers given here and below are extracted during different trials. So they may not correlate.

The Restore Process

It took me 17 mins to run this restore process once, break down as follows:
1) 5 mins - Boot up and restore Linux system image from USB (depending on the speed of your USB device, it may take longer)
2) 2 mins - Uninstalling packages
3) 8 mins - Defragment filesystem [NB. the more files in sda1, i.e. less packages you remove, the longer it will take]
4) 2 mins - Resize filesystem and partition

During the process, you will be prompted by apt-get to confirm removal of programs [Y/n]? for each remove*.list file it processes.
Processing remove_settings.list ...

Reading package lists... Done
Building dependency tree... Done
Note, selecting libclamav-client-perl for regex 'libclamav*'
Note, selecting libclamav-dev for regex 'libclamav*'
Note, selecting libclamav2 for regex 'libclamav*'
Note, selecting libclamav-client-perl for regex 'clamav*'
Note, selecting php5-clamavlib for regex 'clamav*'
Note, selecting clamav for regex 'clamav*'
Note, selecting clamav-freshclam for regex 'clamav*'
Note, selecting clamav-dbg for regex 'clamav*'
Note, selecting libclamav-dev for regex 'clamav*'
Note, selecting clamav-base for regex 'clamav*'
Note, selecting libclamav2 for regex 'clamav*'
Note, selecting sylpheed-claws-clamav for regex 'clamav*'
Note, selecting sylpheed-claws-gtk2-clamav for regex 'clamav*'
Note, selecting clamav-testfiles for regex 'clamav*'
Note, selecting clamav-data for regex 'clamav*'
Note, selecting clamav-getfiles for regex 'clamav*'
Note, selecting python2.4-clamav for regex 'clamav*'
Note, selecting python-clamav instead of python2.4-clamav
Note, selecting clamav-daemon for regex 'clamav*'
Note, selecting clamav-docs for regex 'clamav*'
Note, selecting python-clamav for regex 'clamav*'
Note, selecting clamav-milter for regex 'clamav*'
Note, selecting php4-clamavlib for regex 'clamav*'
The following packages will be REMOVED:
clamav clamav-base clamav-freshclam libclamav2 xandros-antivirus
0 upgraded, 0 newly installed, 5 to remove and 85 not upgraded.
Need to get 0B of archives.
After unpacking 14.3MB disk space will be freed.
Do you want to continue [Y/n]?
[Y/n] means default is "Yes"(Y), you have to press "n" (no) and then [Enter] to cancel. If you simply press [Enter], this means "Yes"(Y).

Later during the process, you will be asked to confirm shrinking the system partition [yes/NO]:
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1 2371820 1363532 887804 63% /mnt

There is now 887804K available disk space in /dev/sda1
Please note that this free space cannot be used by the union filesystem
because /dev/sda1 is always mounted read only.

Do you want to shrink /dev/sda1 to 1558216K? [yes/NO]:
This time, "NO" is in capital letter, meaning default is "NO". If you enter anything other than "yes" or "YES", the script will abort resizing.

So if you answer NO to each prompt, you will end up with the original factory default EeePC Linux, as it has come with your EeePC.

Additional Notes

There are a few points to note during the process:

1) There will be some lines of errors while moving files between sda1 and sda2:
Moving /usr/lib/[b-l] ...
mv: unable to rename '/mnt/usr/lib/libevent.so.1': No such file or directory
mv: unable to rename '/mnt/usr/lib/libgcj_bc.so.1': No such file or directory
mv: unable to rename '/mnt/usr/lib/liblber.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libldap.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libldap_r.so.2': No such file or directory
mv: unable to rename '/mnt/usr/lib/libvolume_id.so': No such file or directory
But I have double-checked that these files (actually symbolic links) mentioned are intact after resizing. So don't worry.

2) With the minimal system, I have found a hissing sound from the speaker as reported many times by other EeeUser forum members (both Linux and winxp). This will not affect music playing, and will go off if you mute the speakers. Since this occur to both Linux and winxp, I am inclined to think it is hardware related (perhaps BIOS?). I have not looked into this yet.

3) The display will blank after 10 minutes of no activity from the keyboard. So it will blank while resizing the partition. Don't panic! Just press any key and display will come back.

4) To defragment the system partition, what I am doing is to copy files from sda1 to sda2 and back again. I break the files into groups of less than 400MB each. This is supposed to be the free disk space in a 2G Surf, so I guess this should be the mininum size of sda2 for different EeePC models.

5) The script will run an "extra.sh" script (in the folder "extra" of the System Recovery USB Disk). It will install "rescue mode" to your grub boot menu, and install your custom initramfs image (if any) and firstrunwizard.conf (to /usr/share/apps/firstrunwizard/) in case you have an recovery iso of a different language version than that you are using.

You will need to edit the "extra.sh" script and give the name of your custom initramfs image and put it in the folder "extra".

6) Theoretically, this restore script should also support restoring to the 2nd SSD (/dev/sdb), but this has not been tested since I don't have a 2nd SSD in my 4G.

7) Likewise, the original restore script will check and detect if the 2nd SSD is non-removable (i.e. not an USB disk) and format it as the HOME partition to be mounted in /home. But there is something missing here: the original script will not set the volume label of sdb1 to HOME if the partition already exist. So many people have reported losing the 16GB partition after USB restore if they have previously installed another Linux in sdb1. I have modified the script to cater for this. But again, since I don't have a 2nd SSD in my 4G, this is untested.

8) The script will save some log files to the folder "log" in the System Recovery USB Disk. This is mainly for debugging and to compare free disk space and fdisk output of partition structure before and after.

The most useful log file will be "removed.log". This log file contains a list of names and version no. of packages removed during this process. If you want to re-install the packages afterwards, this would help giving you the exact package names and version no. [See Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G]

Conclusion

In the end, I got my EeePC Linux version 1.0 back again in my EeePC 4G, still with union filesystem, but most programs can now be upgraded without wasting excessive disk space as now they reside in the USER partition.

For the purpose of testing, I have re-installed all programs I have removed and come back to about the same free disk space as before. It is actually slightly less because of the upgrades.

If it is for real production use, I would probably skip some packages, for example, I won't need the Simplified Chinese locales, and probably also not the games and kids programs. Also, I never use Skype and pidgin ...

If you are more aggressive, there will probably be more packages you can remove provided that you are familiar with Linux system administration. The remove list I have included are all applications that has nothing to do with the base system, that I know has update available, and that their removal will not impair system operation. You can experiment with whatever you like, just make sure the system can still boot up, connect to network, automount USB device, and re-install packages. [Note: Although I have removed firefox, you can still connect to the Internet with the default File Manager. Just type the URL in the address bar as with any browser.]

What's Next?

In the coming few days, I will proceed to test this for different scenarios:
- with different remove lists
- find out the minimum extra disk space %
- test with other iso I have on hand including: 900, 1000H and 1.6 French iso

Hopefully, this script can run for all EeePC Linux versions and for all EeePC models, and be bug free. But for the remove files list, you will have to customize it for yourself. No two users will have the same likings for applications of his or her choice. So it is up to you to decide which packages you want to remove or retain for your EeePC.

Continue:
Part I: My 1st Trial
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system

Thursday, June 4, 2009

Rescue Files from Linux ext2/ext3 partitions

Last night, after restoring my EeePC to the recently downloaded 1.6 iso, I suddenly remembered that my recently written scripts are still in the USER partition (sda2). So I tried to rescue the scripts, and here is what I did (though not successful this time).

There is a free file recovery utility for the Ext2FS/Ext3FS file system called R-Linux which runs in Microsoft Windows. I use to use this to recover files when I accidentally delete some files from my ext2/ext3 partitions of my desktop. So the pre-requisite to use R-Linux is to have a Windows desktop with R-Linux installed. And of course, you should not write anything to the partition you want to recover files from, so you will need to boot from another OS and dump the partition data. What I did is to boot from Pupeee and dd the complete sda2 partition (hdc2 in Pupeee) to a partition in a SD card.
dd if=/dev/hdc2 of=/dev/sdb2 conv=notrunc,noerror
Then I took the SD card to my Windows desktop and scan it with R-Linux.

Here is the "Drive View" at startup of R-Linux. The SD card is recognised as "Disk23", and "Partition1" is the duplicated sda2 from my EeePC.

Next, I select "Disk23" - "Partition1" and click on the [Scan] button.

The program will then scan all inodes for available filesystems. You can see from the screenshot that it will find all filesystems/partitions, some even dated back to (last written) 2008. But only "Recognised0" (the latest modified filesystem) contain valid files that can be recovered.

If you are lucky enough, the files have not been overwritten, then you can simply select them and click on the [Recover] button. The program will then ask you what to save your files, and then you are done.

If you can't find the file you want, you can still use the [Search] function if you can remember the filename. The program also support some advanced search function like searching for last modified date, etc. Basically, R-Linux is very user friendly and useful to recover files from ext2/ext3 partitions.

Wednesday, June 3, 2009

Create an Asus EeePC System Recovery USB disk

Last night, I have downloaded another EeePC Linux 1.6 iso (English version for 701) to try out and as usual I need to first create an Asus EeePC System Recovery USB disk. For me, this procedure is very simple but I don't know why many people have problem doing this. So I take this opportunity to generate some screenshots of the steps to serve as a guidance for others.

My EeePC 4G come with 1 no. Support DVD which contains the P701L.gz Linux system image, winxp drivers, user manual and some utilities. Just put this DVD into the DVDROM of your Windows XP desktop and the tool AsSetup.exe will autorun. [NB. If your Windows have been set to disallow CD autorun, you can navigate into the folder Bin\ and double-click to run it]

From the 2nd tab on the left, you can find the "ASUS Linux USB Flash Utility", with which you can make a USB bootable disk with the Linux system image for EeePC system recovery.

Please note that some EeePC come with 2 DVDs: the Support DVD (with the winxp drivers, manuals and tools), and the Linux Recovery DVD (with the Linux system image). The "ASUS Linux USB Flash Utility" is in the 1st DVD while the Linux system image is in the 2nd DVD. So you may need to swap DVDs during the process to create the Asus EeePC System Recovery USB disk.

If you don't have an original Linux Recovery DVD (e.g. if you have bought the winxp version), there are quite a no of places you can download the iso of this disk. Here are some links I know of:
http://downloads.k0k0.de/index.php?dir=EeePC%2F
http://www.eeepc-logiciels.com/downloads/xandrosFR16
http://www.asustreiber.de/downloads/eeepc/618-images
EeePC 701 ISO on SourceForge.net

In this case, you will also need to download the "ASUS Linux USB Flash Utility" (BootTool.rar) which will help you format and configure a detected USB flash stick to boot, then copy the Linux system image to it.

After you have decompressed the package, you will find a folder BootTool\. Click on any of the .exe files, and you will end up with the same FreeOSTool.exe window.
The screenshots below show the steps how this is run:

If BootTool can detect an USB flash drive, then it should directly arrive at the next screenshot:

Select the 1st line and click on [Run]. You will arrive at a "Confirm Dialog".

Click on [Yes] to continue, and the progress bar will show up.

After formatting is completed, the utility will ask you to unplug and re-plug the USB flash disk, then click on [Retry].

There it will proceed to copy the Linux system image files to the USB flash disk.

Success!


In the end, you should have the System Recovery USB disk with volume label "EEEPC" (or "EEEPC-701" depending on your model), and some folders and files like this in the disk:
total 874M
drwxrwxrwx 3 root root 32K 2007-07-24 16:20 boot/
-r--r--r-- 1 root root 0 2007-10-07 12:53 2007.10.07_04.33.bld
-r--r--r-- 1 root root 6 2007-10-07 12:53 blockcount.dat
-r--r--r-- 1 root root 874M 2007-10-07 12:53 P701L.gz
-r--r--r-- 1 root root 4 2007-10-07 12:53 user_start.dat

./boot:
total 2.9M
drwxrwxrwx 2 root root 32K 2007-09-14 10:55 grub/
-rw-rw-rw- 1 root root 1.6M 2007-10-22 14:57 initrd.gz
-rw-rw-rw- 1 root root 1.3M 2007-06-21 18:49 vmlinuz

./boot/grub:
total 288K
-rw-rw-rw- 1 root root 66 2007-05-23 11:59 device.map
-rw-rw-rw- 1 root root 7.4K 2007-06-19 09:52 e2fs_stage1_5
-rw-rw-rw- 1 root root 7.3K 2007-06-19 09:52 fat_stage1_5
-rw-rw-rw- 1 root root 300 2007-09-14 10:54 menu.lst
-rw-rw-rw- 1 root root 512 2007-06-19 09:52 stage1
-rw-rw-rw- 1 root root 105K 2007-06-19 09:52 stage2
Out of these files, P701L.gz is the Linux system image. It is less than 1GB for 701 and those models with a 4GB 1st SSD. But for those models with a harddisk, e.g. 1000H, it will be as big as 1.5 GB. So for those SSD models, theoretically 1GB should be enough for the System Recovery Disk. This is also what I use to use. But some people have reported problem with 1GB USB flash disks, and now even Asus recommends 2GB flash disk.

Another file of interest is the file /boot/initrd.gz. This is the initial RAM disk image. After the Linux kernel (vmlinuz) is loaded by the grub bootloader, grub will look for, unpack and load this initrd.gz according to the default entry (or the entries you select) in the grub boot menu config file (/boot/grub/menu.lst). This is where the actual bootscript is.

In all those of my hacks: Boot Xandros from USB for Dummies and Restore EeePC with unionfs, what I am doing is just to replace the original bootscript here with a custom bootscript (that loads required kernel modules, seeks the USB flash disk by volume label, mount it in /xan and run another script) and another script that performs all the operations as required.

Some other files, in particular blockcount.dat and user_start.dat, are also vital to the restore process. So check and verify if all the files are here. If you run into problem, you may want to check this wiki: Howto: USB Restore.

If you don't have a Windows XP desktop, you can still do this in Linux. I won't cover it here. Check these wiki:
http://wiki.eeeuser.com/howto:usbrestore#in_linux
HOWTO Make a Standard USB Image under Linux

Now you are ready to restore your EeePC. Plug the System Recovery USB disk into your EeePC, power it up and press [ESC] as soon as you see the Asus BIOS splash screen, then select the USB flash disk as 1st boot device at the BIOS screen similiar to this.

Here is a picture of the screen display during the restore process. You will be prompted to enter "yes" in the middle of the process.

The restore process takes about 5 minutes if you have a fast enough USB flash disk. On some old USB devices, e.g. multi-card reader + mini-SD card, it may take longer. The longest I have experienced is 40 minutes!

P.S. For models with 2 SSD, the bootscript will also format the 2nd SSD to Linux ext3 filesystem and assign a volume label of "HOME" to it. Then the system will mount it to /home at bootup (by seeking volume label). However, what the bootscript does is to look for a non-removable bit in the controller firmware of the 2nd SSD (/dev/sdb). People have reported that after they installed winxp or even ubuntu in their EeePC, the removable bit in the 2nd SSD will be changed to "removable", and the bootscript will think it is a hot-pluggable removable disk and will fail to format it and assign the correct volume label. Then they ended up losing the 16GB 2nd SSD (not automatically mounted by the system).

Monday, June 1, 2009

Restoring my EeePC from USB to a barebone system (Part I - My 1st Trial)

The Story - How it begins

Recently there have been discussion in the EeeUser Forum about a slimmed down version of EeePC Linux or a barebone system. Many people like the F9 restore feature, but want to get rid of some programs they never use.

I don't know how many people are interested in this, but after reading the posts, I am thinking it shouldn't be difficult based on the Boot Xandros from USB for Dummies approach I have been working on. I have seen lazyfai booting from USB and remove some installed programs with his early version of XEPC.org upgrade disc. The only tricky part is to resize the partition afterwards, but I have found a good tutorial on this: How To Resize ext3 Partitions without Losing Data.

Steps in my Original Plan

So after some moments of deep thinking, I think the steps we need to do are:
  1. Boot up from a modified Asus EeePC USB Recovery disk
  2. Seek volume label to mount the recovery disk in /xan
  3. dd the P701L.gz recovery disk image to the 1st SSD
  4. Do the remaining operations as per the original bootscript
  5. Mount /dev/sda1 in /mnt
  6. chroot to /mnt and run apt-get to remove programs you listed in some text files "remove*.list"
  7. Run df to obtain the current System, Used and Available disk size
  8. Calculate the required new size of sda1 + say 5% margin
  9. umount /dev/sda1
  10. Perform disk scan with e2fsck to eliminate any filesystem errors
  11. Shrink the filesystem of sda1 to the calculated required new size with resize2fs
  12. Run e2fsck again to check and eliminate any filesystem errors
  13. Run fdisk to shrink the partition to the new size (make it slightly larger)
  14. Run resize2fs again to expand the filesystem to the partition (adjust for any size difference)
  15. Run fdisk again to expand the USER partition to fill up the free space
  16. Format the USER partition again and set the volume label USER
  17. End of operation: Reboot
For a completely new system restored from USB, this shouldn't be too difficult. But for an old system, it would be very complicated considering the following requirements:
  • Need to let dpkg keep track of changes but dpkg database is spreading over sda1 and sda2
  • Beware of new software installations by the user that have upgraded some newer versions of system libraries now residing in sda2 instead of in sda1.
  • It is not possible to use resize2fs to move the starting disk cylinder of sda2. After you have shrinked sda1, there will be free disk space between sda1 and sda2.

My First Unsuccessful Trial

Last weekend, I proceed to test this out. The first part to remove programs with apt-get is successful. I ended up with 721 MB free disk space in sda1 (versus original 100 MB) after removing openoffice, acrobat reader, Chinese, all games and kids programs:
remove_acroread.list    168 MB *     Acrobat reader
remove_drivers.list 9589 kB Additional hardware drivers
remove_games.list 58.9 MB Programs in PLAY tab / GAMES folder
remove_kids.list 70.5 MB Programs in LEARN tab
remove_ooo.list 301 MB # @ openoffice.org
remove_scim.list 20.9 MB Chinese Input Method
remove_zh_cn.list 41.7 MB * @ Simplified Chinese locales
remove_zh_tw.list 92.6 MB * # Traditional Chinese locales

Remark: Some zh_tw and zh_cn packages have been duplicated in above remove lists
* : include acroread-cn-tw 51.8 MB
# : include openoffice.org-help-zh-tw 27.9 MB
@ : include openoffice.org-help-zh-cn 27.9 MB
The Used size as reported by df is now 1.5 GB (1513604 1K-blocks). I guess this is approaching the minimum size we can anticipate in sda1.

The Need to Defragment

Shrinking the partition however, out of my expectation, is not successful. It is not that it is not possible to shrink the partition with resize2fs, but I have overlooked the need to defragment the filesystem after deleting files belonging to the packages were removed. Although e2fsck is reporting only 0.5% non-contiguous, after resizing, e2fsck reported bad inode references.

Currently there is no available reliable tool in Linux for defragmenting ext2 partitions, because unlike FAT and NTFS, there is rarely the need to defragment Linux ext2/ext3 filesystem. So I am now scratching my head to think it all over again how to get this done. Probably we will need to move all the files to /dev/sda2 and move it back again so as to allow the Linux kernel to automatically defragment them during this process.

Alternative Methods I have done in the Past

Another alternative is to give up on this disk resizing and just install programs into the free disk space of sda1. I have done this a number of times before with slightly different approaches:

The first time, I boot up with no union filesystem (i.e. sda1 only) and upgrade programs from a local repositories based on xepc-upgrade-disc 20080525 . I have written the steps for how to set up a local repos and upgrade the system in this wiki: Upgrade factory OS with the xepc-upgrade-disc. The drawback of this method however, is that after the system boots up without union filesystem, it will write a lot of config and log files into /var, /home/user, etc. of sda1. These files won't be used again when you boot up normally with union filesystem, so just wasting disk space in sda1.

The second time, I tried to reproduce the same result, but working in rescue mode. The result of which is this wiki: Hacking in Rescue Mode. The operation is succesful. But one thing I didn't fully worked out is how to let dpkg keep track of the changes. I had to manually check, compare and copy files from the dpkg database in sda1 (/var/lib/dpkg) to sda2.

The third time, I tested upgrading with a read/write union filesystem as per this blog: Testing read/write union filesystem in my EeePC 4G. But again, this method has its own drawbacks.

So, now I come up with this idea to restore from USB to a barebone system which seems more sensible and more practical.

Continue:
Part II: Success - Restoring my EeePC from USB to a Minimal System
Part III: Re-installing Removed Packages to my minimal EeePC Linux 701 4G
Part IV: 2nd Test restoring to minimal with a 1.6.1 701 en iso
Part V: Yet another test with 1.6 French iso
Conclusion: Restore from USB to minimal system