User Tools

Site Tools


software:psion:file_system_background

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
software:psion:file_system_background [2007/11/22 16:36]
acid_snoop created
software:psion:file_system_background [2008/05/14 17:54] (current)
Line 1: Line 1:
 ====== File System Background ====== ====== File System Background ======
 +
  
 ===== Background ===== ===== Background =====
  
-This covers filesystems basics and the hardware present on the Netbook. The current ​fielsystem ​design document for Netbook LX is [[File System Paper]].+This covers filesystems basics and the hardware present on the Netbook. The current ​filesystem ​design document for Netbook LX is [[software:​psion:​File_System_Paper]]. 
  
 ==== Unix/Linux FS basics ==== ==== Unix/Linux FS basics ====
Line 11: Line 13:
 Unix filesystems are unified - i.e. there is only one '​tree'​ of files and thus only one '​root'​ from which all files are referenced. i.e. this is unlike DOS/​Windows/​EPOC where you first select a device/disk (C: D: Z:) and then navigate into the tree of files below that. In Unix each extra disk/device is '​mounted'​ somewhere in the tree. Doing this overlays anything that was already accessible at that location. In kernel 2.6 this has changed so that it is possible to see both the new files (on the newly mounted disk) and the old files (on the old disk) that appear in the tree at the same place. Unix filesystems are unified - i.e. there is only one '​tree'​ of files and thus only one '​root'​ from which all files are referenced. i.e. this is unlike DOS/​Windows/​EPOC where you first select a device/disk (C: D: Z:) and then navigate into the tree of files below that. In Unix each extra disk/device is '​mounted'​ somewhere in the tree. Doing this overlays anything that was already accessible at that location. In kernel 2.6 this has changed so that it is possible to see both the new files (on the newly mounted disk) and the old files (on the old disk) that appear in the tree at the same place.
  
-Although ​everytihg ​appears as one tree, each device can have it's own filesystem, so quite complex arrangements of storage types and characteristics can be set up. This is one reason why unix installs files by their type of use (config files into /etc, user files into /home, temporary files into /tmp local binaries into /bin) so these could potentially be on different filesystems/​partitions on different disks, in ram, or over the network, shared between many machines. This works well except where you want to store a whole set of files for different applications on a different medium -then it works against you.+Although ​everything ​appears as one tree, each device can have it's own filesystem, so quite complex arrangements of storage types and characteristics can be set up. This is one reason why unix installs files by their type of use (config files into /etc, user files into /home, temporary files into /tmp local binaries into /bin). So these could potentially be on different filesystems/​partitions on different disks, in ram, or over the network, shared between many machines. This works well except where you want to store a whole set of files for different applications on a different medium -then it works against you.
  
 ===== Hardware Options ===== ===== Hardware Options =====
Line 31: Line 33:
  
 As we have quite a lot of RAM on the Netbook Lx but are short of flash it seems that JFFS2'​s compression is much more important than its overheads so this seems likely to be the best Filesystem. It is widely used in portable devices and has been very well tested, although the NAND support is relatively new (2 years). There should be no problems using this. As we have quite a lot of RAM on the Netbook Lx but are short of flash it seems that JFFS2'​s compression is much more important than its overheads so this seems likely to be the best Filesystem. It is widely used in portable devices and has been very well tested, although the NAND support is relatively new (2 years). There should be no problems using this.
 +
  
 ==== Compact Flash cards ==== ==== Compact Flash cards ====
  
-Compact flash is a slightly confusing technology. It looks like IDE (hard drives) to the machine and can be used as a direct replacement,​ but inside the device is flash. This used to be NOR flash, but these days is NAND flash. A controller is included in the device to convert between IDE (ATA commands) and the flash chips. We have no control over this controller and manufacturers don't usually tell you much about what it does. The important things from our point of view is how the controller distributes writes over the flash. If it just uses a direct mapping from disk sector to flash page then filesystems that repeatedly write to the same area will wear out those flash zones quickly.The controllers have a stack of '​spare'​ flash which they use to fill holes in the device when sectors go bad. It is a feature of NAND flash that bad sectors are expected, even from new. Smarter controllers should attempt to distibute ​writes round the device, and it would be smart to see if it is worth buying devices that do this. However we do not have control of the CF cards people use so need to try and minimise ​any problems of flash wear by filesystem choice and overall system design.+Compact flash is a slightly confusing technology. It looks like IDE (hard drives) to the machine and can be used as a direct replacement,​ but inside the device is flash. This used to be NOR flash, but these days is NAND flash. A controller is included in the device to convert between IDE (ATA commands) and the flash chips. We have no control over this controller and manufacturers don't usually tell you much about what it does. The important things from our point of view is how the controller distributes writes over the flash. If it just uses a direct mapping from disk sector to flash page then filesystems that repeatedly write to the same area will wear out those flash zones quickly.The controllers have a stack of '​spare'​ flash which they use to fill holes in the device when sectors go bad. It is a feature of NAND flash that bad sectors are expected, even from new. Smarter controllers should attempt to distribute ​writes round the device, and it would be smart to see if it is worth buying devices that do this. However we do not have control of the CF cards people use so need to try and minimize ​any problems of flash wear by filesystem choice and overall system design.
  
-Because the CF card appears as an IDE device (i.e through the block layer) it if difficult to use filesystems designed explicitly for NAND flash - we have to use filesystems designed for disks. There is a module called blkmtd which allows an IDE device to be accessed as if it was a real MTD device, which is normally a crazy thing to do but may make sense for CF cards. This can be used with JFFS2, but not with YAFFS (because the OOB information for the flash chips is not exposed). This will improve wear-levelling, but cannot deal with flash errors.+Because the CF card appears as an IDE device (i.e through the block layer) it if difficult to use filesystems designed explicitly for NAND flash - we have to use filesystems designed for disks. There is a module called blkmtd which allows an IDE device to be accessed as if it was a real MTD device, which is normally a crazy thing to do but may make sense for CF cards. This can be used with JFFS2, but not with YAFFS (because the OOB information for the flash chips is not exposed). This will improve wear-leveling, but cannot deal with flash errors.
  
-We are also constrained if we want to be able to boot from the CF with the existing BooSt - this only recognises ​a FAT filesystem so any card which we can boot from needs a FAT partition at the start of the device. This may only be necessary during development or for system restore. The same consideration affects transfer of files to other devices that read CF cards - nearly all will expect to see VFAT.+We are also constrained if we want to be able to boot from the CF with the existing BooSt - this only recognizes ​a FAT filesystem so any card which we can boot from needs a FAT partition at the start of the device. This may only be necessary during development or for system restore. The same consideration affects transfer of files to other devices that read CF cards - nearly all will expect to see VFAT.
  
-Our exerience ​of ext3 on the Samdisk CF cards shows that these devices do the above simple mapping and because of the way the ext3 journalling ​works (all writes go through a 32Mb section at the end of the drive), this part of the device wears out extremely fast if atime modification is enabled (a few days/​reboots). However with atime updates disabled it is probably a viable choice, although the journalling ​does increase the number of writes to the device.+Our experience ​of ext3 on the Samdisk CF cards shows that these devices do the above simple mapping and because of the way the ext3 journaling ​works (all writes go through a 32Mb section at the end of the drive), this part of the device wears out extremely fast if atime modification is enabled (a few days/​reboots). However with atime updates disabled it is probably a viable choice, although the journaling ​does increase the number of writes to the device.
  
 Common filesystems for disk use are: vfat, ext2, ext3, reiserfs, jfs, xfs, ntfs.  Common filesystems for disk use are: vfat, ext2, ext3, reiserfs, jfs, xfs, ntfs. 
  
-Vfat and ntfs are not native UNix FSes and do not support important features like user and group IDs, access permissions and links so are not practical for most systems. Linux NTFS write support is also not well tested. We can ignore these for storing unix system files, but VFAT is useful for storing files that need to be read on other devices and compatibility(cameras, ​Boo'​St ​loading etc.). VFAT is famously unreliable as a frequently-updated filing system.+Vfat and ntfs are not native UNix FSes and do not support important features like user and group IDs, access permissions and links so are not practical for most systems. Linux NTFS write support is also not well tested. We can ignore these for storing unix system files, but VFAT is useful for storing files that need to be read on other devices and compatibility(cameras, ​BooSt loading etc.). VFAT is famously unreliable as a frequently-updated filing system.
  
 Ext2 is very reliable and well tested. It's main problem is that it must be shut down properly otherwise it has to do automatic file integrity checks on startup. ('​fsck'​ing). This can take a long time (3 minutes on 512Mb CF?). The importance of this depends on how often the system is going to be rebooted without having gone through a proper shutdown. As a battery-powered device it seems that this is quite likely to happen so this filesystem is not ideal. It also makes no special efforts to distribute writes, although it doesn'​t concentrate writes at the start of the device as much as FAT does [Is this true?]. Ext2 is very reliable and well tested. It's main problem is that it must be shut down properly otherwise it has to do automatic file integrity checks on startup. ('​fsck'​ing). This can take a long time (3 minutes on 512Mb CF?). The importance of this depends on how often the system is going to be rebooted without having gone through a proper shutdown. As a battery-powered device it seems that this is quite likely to happen so this filesystem is not ideal. It also makes no special efforts to distribute writes, although it doesn'​t concentrate writes at the start of the device as much as FAT does [Is this true?].
  
-Ext3, reiserfs, jfs and xfs are all journalled ​so they do not lose filesystem integrity on sudden power loss (unwritten bits of data may be lost of course). However these journals are not-necessarily done the same way as the flash filesystem journals in JFFS2 and YAFFS, so they don't necessarily reduce flash wear.+Ext3, reiserfs, jfs and xfs are all journaled ​so they do not lose filesystem integrity on sudden power loss (unwritten bits of data may be lost of course). However these journals are not-necessarily done the same way as the flash filesystem journals in JFFS2 and YAFFS, so they don't necessarily reduce flash wear.
  
 Ext3 uses a 32MB journal at the end of the drive and all writes go through this section, then are rewritten to the right place later (by kjournald) so in fact it concentrates all writes in one part of the device and doubles the number of writes overall. Ext3 uses a 32MB journal at the end of the drive and all writes go through this section, then are rewritten to the right place later (by kjournald) so in fact it concentrates all writes in one part of the device and doubles the number of writes overall.
  
-Reiserfs uses a very different strategy of balanced B-trees or (in reiser4) '​dancing trees'​. It stores small files very efficiently and as the oldest ​journalled ​FS (reiser3) is very reliable. There is a new Reiser4 version out which is even faster and more wonderful, but very new. I still haven'​t found out how either version distributes writes though, and thus whether it would really help.+Reiserfs uses a very different strategy of balanced B-trees or (in reiser4) '​dancing trees'​. It stores small files very efficiently and as the oldest ​journaled ​FS (reiser3) is very reliable. There is a new Reiser4 version out which is even faster and more wonderful, but very new. I still haven'​t found out how either version distributes writes though, and thus whether it would really help.
  
 JFS and XFS are both designed for large (datacenter style) storage and thus probably don't offer useful advantages. JFS and XFS are both designed for large (datacenter style) storage and thus probably don't offer useful advantages.
software/psion/file_system_background.txt · Last modified: 2008/05/14 17:54 (external edit)