JM2L
Updateon filesystems forflashstorage
MichaelOpdenacker. FreeElectrons http://freeelectrons.com/
1
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Contents
Introduction Availableflashfilesystems Ourbenchmarks Bestchoices Experimentalfilesystems Adviceforflashbasedblockdevices
2
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Updateonfilesystemsforflashstorage
Introduction
3
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Flashstorage
Wearetalkingaboutflashchips,accessedbytheLinux kernelasMemoryTechnologydevices. CompactFlash,MMC/SD,MemoryStickcards,together withUSBflashdrivesandSolidStateDrives(SSD), areinterfacedasblockstorage,likeregularharddisks. Attheend,wewillsayafewwords aboutdealingwiththesecondcategory.
4
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Existingsolutions
Forthelastyears,only2filesystemchoicesforflashstorage jffs2 Wearleveling,ECC Powerdownresistant Compression Hugemounttimes Ratherbigmemoryusage Mainstreamsupport yaffs2 Wearleveling,ECC Powerdownresistant Nocompression Veryquickmounttime ProgrammedbyWookies (atleast1) AvailableasaLinuxpatch. 2solutions,butfarfrombeingperfect!
5
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Electiontime!
Atlast,newchoiceshavebeendeveloped. LogFS NewfilesystemforMTD storage UBI Newlayermanagingerase blocksandwearleveling UBIFS Newfilesystemtaking advantageofUBI's capabilities AXFS AdvancedXIPFileSystem
6
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Howdotheycompare toexistingsolutions? Mountingtime Accessspeed Memoryusage CPUusage Size?
Testhardware
CalaoSystemsUSBA9263 AT91SAM9263ARMCPU 64MBRAM 256MBflash 2USB2.0host 1USBdevice 100MbitEthernetport PoweredbyUSB! SerialandJTAGthrough thisUSBport. Multipleextensionboards. 162EUR SupportedbyLinux2.6.27!
7
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Flashchips
NANDdevice:ManufacturerID:0xec,ChipID:0xda (SamsungNAND256MiB3,3V8bit)
Samsung'sreference:K4S561632HUC75
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Updateonfilesystemsforflashstorage
Availableflashfilesystems
9
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
TheMTDAPI
ALinuxkernelAPItoaccessMemoryTechnologyDevices AbstractsthespecificsofMTDdevices:eraseblocks,pagesize... Linuxfilesysteminterface MTDUsermodules
jffs2 yaffs2 Chardevice Readonlyblockdevice Blockdevice
MTDChipdrivers
CFIflash NANDflash DiskOnChipflash
RAMchips ROMchips
Memorydeviceshardware
10
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
MTDHowtouse
Creatingthedevicenodes Chardevicefiles mknod/dev/mtd0c900(badidea!!!) mknod/dev/mtd1c902(Caution!) mknod/dev/mtd2c904(Caution) Blockdevicefiles mknod/dev/mtdblock0b310(badidea!!!) mknod/dev/mtdblock0b312 mknod/dev/mtdblock0b312
11
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
jffs2
Today'sstandardfilesystemforMTDflash Nicefeatures:
Ontheflycompression.Savesstoragespace andreducesI/O. Powerdownreliable. Implementswearleveling
Standardfile API JFFS2 filesystem MTDdriver
Drawbacks:doesn'tscalewell
Mounttimedependingonfilesystemsize: thekernelhastoscanthewholefilesystemat mounttime,toreadwhichblockbelongsto eachfile. KeepingthisinformationinRAMismemory hungrytoo.
Flashchip
12
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Newjffs2features
CONFIG_JFFS2_SUMMARY Reducesboottimebystoringsummaryinformation. Newjffs2compressionoptions:
Nowsupportslzocompression,andnotonlyzlib (andalsothertimeandrubincompressors) Cantryallcompressorsandkeep theonegivingthebestresults Canalsogivepreferencetolzo,totheexpenseofsize, becauselzohasthefastestdecompressiontimes.
13
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
jffs2Howtouse
Compilemtdtoolsifneeded: gitclonegit://git.infradead.org/mtdutils.git Eraseandformatapartitionwithjffs2: flash_eraseallj/dev/mtd2 Mountthepartition: mounttjffs2/dev/mtdblock2/mnt/flash Fillthecontentsbywriting Or,useanimage: nandwritep/dev/mtd2rootfs.jffs2
14
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
yaffs2
http://www.yaffs.net/ SupportsbothNANDandNORflash Nocompression Wearleveling,ECC,powerfailureresistant Fastboottime CodeavailableseparatelythroughCVS (DualGPL/Proprietarylicense fornonLinuxoperatingsystems) YAFFS2 filesystem MTDdriver Standardfile API
Flashchip
15
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
yaffs2Howtouse
Eraseapartition: flash_eraseall/dev/mtd2 Formatthepartition: sleep(any command can do!) Mountthepartition: mounttyaffs2/dev/mtdblock2/mnt/flash
16
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
UBI
UnsortedBlockImages http://www.linuxmtd.infradead.org/doc/ubi.html VolumemanagementsystemontopofMTDdevices. Allowstocreatemultiplelogicalvolumes andspreadwritesacrossallphysicalblocks. Takescareofmanagingtheeraseblocksandwearleveling. Makesfilesystemeasiertoimplement.
UBI Logical EraseBlocks MTD Physical EraseBlocks Volume1 Volume2
LEB
LEB
LEB
LEB
LEB
LEB
LEB
PEB PEB PEB PEB PEB PEB PEB PEB PEB
Freeblock Freeblock 17
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
UBIHowtouse(1)
First,eraseyourpartition(NEVERFORGET!) flash_eraseall/dev/mtd1 First,formatyourpartition: ubiformat/dev/mtd1s512 (possibletosetaninitialerasecountervalue) Seehttp://www.linuxmtd.infradead.org/faq/ubi.htmlifyoufaceproblems Needtocreatea/dev/ubi_crtldevice(ifyoudon'thaveudev) Majorandminornumberallocatedinthekernel.Findthesenumbersin /sys/class/misc/ubi_ctrl/dev/(e.g.:10:63) Orrunubinfo: UBIversion:1 CountofUBIdevices:1 UBIcontroldevicemajor/minor:10:63 PresentUBIdevices:ubi0
18
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
UBIHowtouse(2)
AttachUBItoone(ofseveral)oftheMTDpartitions: ubiattach/dev/ubi_ctrlm1 FindthemajorandminornumbersusedbyUBI: cat/sys/class/ubi/ubi0/dev(e.g.253:0) CreatetheUBIdevicefile: mknod/dev/ubi0c2530
19
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
UBIFS
http://www.linuxmtd.infradead.org/doc/ubifs.html Thenextgenerationofthejffs2filesystem, fromthesamelinuxmtddevelopers. AvailableinLinux2.6.27 WorksontopofUBIvolumes UBI MTDdriver Standardfile API UBIFS
Flashchip
20
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
UBIFSHowtouse
Creating ubimkvol/dev/ubi0Ntests116MiB mounttubifsubi0:test/mnt/flash Deleting umount/mnt/flash ubirmvol/dev/ubi0Ntest DetachtheMTDpartition: ubidetach/dev/ubi_ctrlm1
21
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
LogFS
http://logfs.org/logfs/ Alsodevelopedasareplacementforjffs2 Weannouncedwewouldcoverit,butitslatestversiononly supports2.6.25.Ourboardonlysupports2.6.21,2.6.27and beyond,andthe2.6.25LogFSpatchdoesn'tcompilein 2.6.27! Anyway,LogFSisnotreadyyetforproduction. Williteverbe,nowthatjffs2hasavaluablereplacement? Competitionisusefulthough.
22
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
AXFS
AdvancedXIPFileSystemforLinux http://axfs.sourceforge.net/ Allowstoexecutecodedirectlyfromflash, insteadofcopyingittomemory. AsXIPisnotpossiblewithNANDflash,worksbestwhen thereisamixofNORflash(forcode)andNAND(fornon XIPsections). Currentlypostedforreview/inclusioninthemainstream Linuxkernel.Tobeacceptedin2.6.29orlater? Notbenchmarkedhere.WeonlyhaveNANDflashanyway.
23
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
SquashFS
http://squashfs.sourceforge.net/ Filesystemforblockstorage!? Butreadonly!Noproblemwithmanagingeraseblocksand wearleveling.Finetousewiththemtdblockdriver. Youcanuseitforthereadonlysectionsinyourfilesystem. Activelymaintained.Releasesformanykernelversions (recentandold).
24
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
SquashFSHowtouse
Verysimple! Onyourworkstation,createyourfilesystemimage (example:120m/directoryinourbenchmarks) mfsquashfs120m120m.sqfs Eraseyourflashpartition: flash_eraseall/dev/mtd2 Makeyourfilesystemimageavailabletoyourdevice(NFS, copy,etc.)andflashyourpartition: ddif=120m.sqfsof=/dev/mtdblock2 Mountyourfilesystem: mounttsquashfs/dev/mtdblock2/mnt/flash
25
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Updateonfilesystemsforflashstorage
Benchmarks
26
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Benchmarkoverview
Comparedfilesystems: jffs2,defaultoptions jffs2,lzocompressiononly yaffs2 ubifs,defaultoptions ubifs,nocompression squashfs DifferentMTDpartitions 8M 32M 120M Correspondingtomost embeddeddevicescenarios. Partitionsfilledatabout85%
AlltestedwithLinux2.6.27.
27
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Readandmountingexperiments
MountinganarmLinuxrootfilesystem, takenfromtheOpenMokoproject. Advantages:mainlycontainscompressiblefiles (executablesandsharedlibraries). Representsaveryimportantscenario:bootingona filesysteminflash.Mountingandfileaccesstimearemajor componentsofsystemboottime.
28
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Mounttime(seconds)
ubifsnoz/8M:doesn'tfit
18 16
14
12
10 jffs2 jffs2-lzo yaffs2 ubifs ubi-noz squashfs
0 8M 32M 120M
29
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
ZoomMounttime(seconds)8M
ubifsnoz/8M:doesn'tfit
1.4
1.2
0.8 jffs2 jffs2-lzo yaffs2 ubifs ubi-noz squashfs
0.6
0.4
0.2
0 8M
30
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Memoryconsumptionaftermounting(KB)
Freememorymeasuredwith/proc/meminfo: MemFree+Buffers+Cached
1400
Nomistake. Proportional tofssize?
1200
1000
800
600
jffs2 jffs2-lzo yaffs2 ubifs ubi-noz squashfs
400
200
0 8M 32M 120M
31
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Usedspace(MB)
Measuredwithdf AddsomespaceforUBIFS! 1MBfor8MB
120
100
80
60
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz squashfs
40
20
0 8M 32M 120M
32
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Readtime(seconds)
140
120
100
80 jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz squashfs
60
40
20
0 8M 32M 120M
33
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
ZoomReadtime(seconds)8M
8
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz squashfs
0 8M
34
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
CPUusageduringread(seconds)
Duringtheexperimentsinthepreviousslide (usingthesysmeasurefromthetimecommand)
120
100
80
60
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz squashfs
40
20
0 8M 32M 120M
35
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Fileremovaltime(seconds)
Removingallthefilesinthepartition (afterthereadexperiment)
45
40
35
30
25 jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
20
15
10
0 8M 32M 120M
36
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Writeexperiment
Writing8Mdirectorycontentsmultipletimes (lessinthe8Mcase) Datacopiedfromatmpfsfilesystem, fornooverheadreadingthefiles. Contents:armLinuxrootfilesystem. Smalltomediumsizefiles,mainlyexecutablesandshared libraries. Notmanyfilesthatcan'tbecompressed.
37
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Writetime(seconds)
yaffs2/8M32M120M:doesn'tfit ubifsnoz/8M:doesn'tfit
300
250
200
150
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
100
50
0 8M 32M 120M
38
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
ZoomWritetime(seconds)8M
yaffs2/8M:doesn'tfit ubifsnoz/8M:doesn'tfit
18
16
14
12
10 jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
0 8M
39
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
CPUusageduringwrite(seconds)
Duringtheexperimentsinthepreviousslide (usingthesysmeasurefromthetimecommand)
300
250
200
150
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
100
50
0 8M 32M 120M
40
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Randomwriteexperiment
Writing1MBchunksofrandomdata (copiedfrom/dev/urandom). Tryingtomimicthebehaviorofdigitalcamerasand camcorders,recordingalreadycompresseddata.
41
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Randomwritetime(seconds)
Caution:includesCPUtimegeneratingrandomnumbers!
1000
900
800
700
600
500
400
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
300
200
100
0 5M 27M 105M
42
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
ZoomRandomwritetime(seconds)8M
Caution:includesCPUtimegeneratingrandomnumbers!
40
35
30
25
20
jffs2 jffs2-lzo yaffs2 ubifs ubifs-noz
15
10
0 5M
43
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Otherexperiments
UBIFSwithonlylzosupport UBIFSsupportsbothlzo(fastertocompressand uncompress)andzlib(slower,butcompressesbetter),and triestofindthebestspeed/sizecompromise. WetriedUBIFSwithonlylzosupport,hopingthathaving onlyonecompressorwouldreduceruntime. Results:tinydifferencesinallbenchmarks,eveninCPU usage.(roughlybetween0.1and1%). Conclusion:don'ttrytobetoosmart. Thefilesystemisalreadyfinetuned toworkgreatinmostcases.
44
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Suitabilityforverysmallpartitions
8MMTDpartition jffs2fits13MBoffiles Butprobablydoesn'tleave enoughfreeblocks UBIconsumes0.9MB ubifsfits6.6MBoffiles 4MMTDpartition jffs2fits5.1MBoffiles UBIconsumes0.8MB ubifsfitsonly1.6MBof files!
Biggersizes:UBIoverheadcanbeneglected: 32MB:consumes1.2MB 128MB:consumes3.6MB
45
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Whatweobserved
jffs2 Dramaticallyoutperformedby ubifsinmostaspects. Hugemount/boottime. yaffs2 Alsooutperformedbyubifs. Maynotfitallyourdata Uglyfileremovaltime (poordirectoryupdate performance?) Memoryusagenotscaling ubifsleavesnoreason tosticktoyaffs2.
46
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
ubifs Greatperformanceinall cornercases. SquashFS Bestornearbest performance inallreadonlyscenarios.
Conclusions
Convertyourjffs2partitionstoubifs! Itmayonlymakesensetokeepjffs2 forMTDpartitionssmallerthan10MB, incasesizeiscritical. Noreasonlefttouseyaffs2insteadofjffs2? YoumayalsouseSquashFStosqueezemore stuffonyourflashstorage.Advisabletouseiton topofUBI,toletallflashsectorsparticipateto wearleveling.
SquashFS MTDblock MTDAPI UBI MTDdriver Flashchip
47
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Experimentalfilesystems(1)
Alookatpossiblefuturesolutions? wikifs ACELFsponsoredproject. AWikistructuredfilesystem (today'sflashfilesystems arelogstructured). AlreadyusedinSonydigital camerasandcamcorders. Pros:direct/easyexportof devicefunctionality descriptiontoelinux.org. linuxtinyfs Targetssmallembedded systems. Negativememory consumption:achievedby compilingoutthekernelfile cache. Pros:veryfastmounttime Cons:amountonly filesystem.Wayto implementreadandwrite notfoundyet.
48
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Experimentalfilesystems(2)
fsckfs Aninnovativefilesystemrebuildingitselfateachreboot. Pros:nouserspacetoolsareneeded. Nofsck.fsckfsutilityneeded. Cons:mounttimestillneedsimproving.
49
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Updateonfilesystemsforflashstorage
Adviceforflashbasedblockstorage
50
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Issueswithflashbasedblockstorage
Flashstoragemadeavailableonlythroughablockinterface. Hence,nowaytoaccessalowlevelflashinterface andusetheLinuxfilesystemsdoingwearleveling. Nodetailsaboutthelayer(FlashTranslationLayer)they use.Detailsarekeptastradesecrets,andmayhidepoor implementations. Hence,itishighlyrecommendedtolimitthenumberof writestothesedevices.
51
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Reducingthenumberofwrites
Mountyourfilesystemsasreadonly,orusereadonly filesystems(SquashFS),wheneverpossible. KeepvolatilefilesinRAM(tmpfs) Usethenoatimemountoption,toavoidupdatingthe filesystemeverytimeyouaccessafile.Oratleast,ifyou needtoknowwhetherfileswerereadaftertheirlastchange, usetherelatimeoption. Don'tusethesyncmountoption (commitswritesimmediately).Nooptimizationspossible. Youmaydecidetodowithoutjournaledfilesystems. Theycausemorewrites,butarealsomuchmorepower downresistant.
52
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Usefulreading
IntroductiontoJFFS2andLogFS: http://lwn.net/Articles/234441/ NiceUBIpresentationfromToshiba: http://freeelectrons.com/redirect/celfubi.html Documentationonthelinuxmtdwebsite: http://www.linuxmtd.infradead.org/
53
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
Updateonfilesystemsforflashstorage
Documentonhttp://freeelectrons.com/pub/conferences CreativeCommonsBYSAlicense
Thankyou!
Questions? Newfilesystem suggestions?
54
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com
