Debian GNU / Linux kernel tuning for low latency on small files

Debian GNU / Linux kernel tuning for low latency on small files

- janeiro 03, 2018

E-mail servers, data base servers, file servers, web servers struggle to cope with LATENCY on accessing millions of small files under heavy concurrent load.
Low latency is key factor to a good user experience.
With the Debian GNU / Linux 6.x kernel tuning, combined with the previous article hints about XenServer I/O latency, tuning filesystems configuring multipath, and data storage, we reduced eighty-four times our virtual block device latency at our environment.

E-mail servers, data base servers, file servers, web servers struggle to cope with LATENCY on accessing millions of small files under heavy concurrent load.
Low latency is key factor to a good user experience.
With the Debian GNU / Linux 6.x kernel tuning, combined with the previous article hints about XenServer I/O latency, tuning filesystems configuring multipath, and data storage, we reduced eighty-four times our virtual block device latency at our environment.
Thousands concurrent synchronous writes of random small files on filesystems with millions of them while honoring write barriers for POSIX compliance, stress the limits of the data storage, file systems, kernel I/O, virtualization I/O, hardware data paths.
These are our findings and Linux kernel tunings in Cyrus IMAP e-mail servers grid under heavy load production since November 2011.
First of all, do not blindly apply these settings at your servers. Read the whole bibliography to understand what you are about to tune.
Understand what each parameter is doing.
Understand your server workload profile pattern.
Understand your hardware and storage performance and behaviour.
Understand your Fiber Channel SAN or iSCSI data path and network segment behaviour.
OUR environment has servers using Fiber Channel HBA multipath connected to FC disks WAFL Data Storage with very low latency and lots of ECC NVRAM for cache. It almost behaves as a giant SSD storage from latency standpoint.
This is important: if you are going to “flush” your servers kernel buffers as fast they could, you data storage and data path should be able to cope with such high IOPS and trough output.
You could even get out -of -memory errors caused by high latency storage hardware. The linked article author gone the opposite direction in tuning due to HIS hardware environment.
Read YOUR deployed kernel version documentation. The links below are for the latest kernel.org version.
http ://www .kernel .org /doc /Documentation /block /queue -sysfs .txt
http ://www .kernel .org /doc /Documentation /block /deadline -iosched .txt
http ://www .kernel .org /doc /Documentation /sysctl /vm .txt
http ://www .nextre .it /oracledocs /ioscheduler _01.html
Also, the net result is only as fast as your slowest link at your application stack, hardware and network stacks.
Carefully configure your application stack for low latency and performance.
At other articles we covered filesystem tuning, data storage LUN configurations, and some aspects of Cyrus IMAP configurations. More on future articles about performance tuning.
You will carefully watch the iostat output during the tests, looking for the bottlenecks.
Read the previous article bibliography and test results about XenServer I/O latency.
Learn how to correlate the output columns with the kernel parameters and your data storage behaviour. Read the manpage of iostat.
iostat -dmxthN 1 /dev/xvd*

Non persistent between reboots configuration
Use these during your test, tuning and evaluation phase.
Logged as root:
sysctl vm.swappiness=10
sysctl vm.dirty_background_ratio=1
sysctl vm.dirty_expire_centisecs=500
sysctl vm.dirty_ratio=15
sysctl vm.dirty_writeback_centisecs=100
cat /sys/block/xvdb/queue/nr_requests
echo "4" > /sys/block/xvdb/queue/nr_requests
cat /sys/block/xvdb/queue/nr_requests
cat /sys/block/xvdb/queue/scheduler
echo deadline > /sys/block/xvdb/queue/scheduler
cat /sys/block/xvdb/queue/scheduler
cat /sys/block/xvdb/queue/iosched/front_merges
echo 1 >/sys/block/xvdb/queue/iosched/front_merges
cat /sys/block/xvdb/queue/iosched/front_merges
cat /sys/block/xvdb/queue/iosched/fifo_batch
echo 1 >/sys/block/xvdb/queue/iosched/fifo_batch
cat /sys/block/xvdb/queue/iosched/fifo_batch

A very aggressive configuration (watch your %sys during test phase):
sysctl vm.dirty_expire_centisecs=50
sysctl vm.dirty_writeback_centisecs=10
sysctl vm.dirty_background_ratio=0

Debian GNU/Linux configuration persistent between reboots
You will need the “sysfsutils” package installed.

Append to your /etc/sysfs.conf
#AFM 20120523
block/xvdb/queue/nr_requests = 4
block/xvdb/queue/scheduler = deadline
block/xvdb/queue/iosched/front_merges = 1
block/xvdb/queue/iosched/fifo_batch = 1

Append to your /etc/sysctl.conf
#AFM 20120523
vm.swappiness = 10
vm.dirty_background_ratio = 1
vm.dirty_expire_centisecs = 500
vm.dirty_ratio = 15
vm.dirty_writeback_centisecs = 100

Some default values for the deadline i/o scheduler
For Debian GNU / Linux 6.x Squeeze, kernel 2.6.32
cat /sys/block/xvdb/queue/iosched/fifo_batch
16
cat /sys/block/xvdb/queue/iosched/front_merges
1
cat /sys/block/xvdb/queue/iosched/read_expire
500
cat /sys/block/xvdb/queue/iosched/write_expire
5000
cat /sys/block/xvdb/queue/iosched/writes_starved
2

Bibliography

kswapd0 , ksoftirqd0
http ://forums .citrix .com /message .jspa ?messageID =1511105 ************************** tuning de kernel para arquivos pequenos
http ://askubuntu .com /questions /7858/why -is -ksoftirqd -0-process -using -all -my -cpu *
http ://askubuntu .com /questions /73639/what -are -kswapd 0-kworker -numnum -ksoftirqd -num *
http ://www .mail -archive .com /rhelv 5-list @redhat .com /msg 05345.html
http ://lkml .indiana .edu /hypermail /linux /kernel /0506.0/1150.html
http ://grokbase .com /t /centos .org /centos /2011/05/centos -kswapd -taking -100-cpu -with -no -swap -on -system /07 g 622 yntfakij 4 lsruvrqxyjeom
http ://answers .softpicks .net /answers /topic /High -CPU -load -with -kswapd -and -heavy -disk -I -O -1747240-1.htm
https ://bugzilla .redhat .com /show _bug .cgi ?id =115438
http ://www .spinics .net /lists /linux -mm /msg 16292.html

swappiness
http ://feedblog .org /2006/09/27/stupid -linux -swap -tricks -with -swappiness /
http ://www .linuxvox .com /2009/10/what -is -the -linux -kernel -parameter -vm -swappiness /
https ://help .ubuntu .com /community /SwapFaq
http ://kerneltrap .org /node /3000
http ://lists .centos .org /pipermail /centos /2011-May /111433.html
https ://bugzilla .redhat .com /show _bug .cgi ?id =135312
https ://bugs .launchpad .net /ubuntu /+bug /721896
https ://bugzilla .kernel .org /show _bug .cgi ?id =12309
http ://us .generation -nt .com /kswapd -causing -giant -load -help -179939921.html
http ://www .linuxquestions .org /questions /linux -server -73/100-swap -usage -with -loads -of -free -memory -849452/
https ://www .centos .org /modules /newbb /viewtopic .php ?topic _id =28555
https ://bugzilla .redhat .com /show _bug .cgi ?id =437202
http ://kerneltrap .org /mailarchive /linux -kernel /2007/9/24/294490
http ://www .linuxforums .org /forum /kernel /65380-what -does -kswapd 0-do .html

Linux kernel tuning for low latency block devices and small files
http ://www .cyrius .com /debian /nslu 2/linux -on -flash .html
http ://www .kernel .org /doc /Documentation /sysctl /vm .txt ****************
http ://serverfault .com /questions /126413/limit -linux -background -flush -dirty -pages ******
http ://www .westnet .com /~gsmith /content /linux -pdflush .htm ****************
http ://www .redhat .com /archives /linux -lvm /2004-February /msg 00115.html ****
http ://www .fccps .cz /download /adv /frr /hdd /hdd .html #tuning -intro ***************
http ://www .serverphorums .com /read .php ?12,115991
http ://www .r 71.nl /kb /technical /102-tuning -journaling -file -systems
http ://support .citrix .com /article /CTX 127065
http ://forums .citrix .com /thread .jspa ?messageID =1487300
http ://knol .google .com /k /linux -performance -tuning -and -measurement *********
http ://kerneltrap .org /node /4462 ***
http ://www .mjmwired .net /kernel /Documentation /block /barrier .txt *****
http ://en .wikipedia .org /wiki /Slab _allocation *
http ://www .ibm .com /developerworks /linux /library /l -linux -slab -allocator /
http ://www .win .tue .nl /~aeb /linux /lk /lk -9.html **
http ://www .fccps .cz /download /adv /frr /hdd /hdd .html ****
http ://makarevitch .org /rant /raid /
http ://www .softpanorama .org /Admin /performance _monitoring .shtml
http ://www .linuxinsight .com /proc _sys _vm _vfs _cache _pressure .html *******
http ://rudd -o .com /en /linux -and -free -software /tales -from -responsivenessland -why -linux -feels -slow -and -how -to -fix -that
https ://bbs .archlinux .org /viewtopic .php ?id =40111
http ://duopetalflower .blogspot .com /2009/11/tuning -ubuntu -910-karmic -for -speed -and .html
http ://www .pythian .com /news /247/basic -io -monitoring -on -linux / ***
http ://www .informit .com /articles /article .aspx ?p =481867 ***
http ://bhavin .directi .com /iostat -and -disk -utilization -monitoring -nirvana / ****
http ://www .xaprb .com /blog /2009/08/23/how -to -find -per -process -io -statistics -on -linux /
http ://www .faqs .org /docs /linux _admin /buffer -cache .html ***********
http ://docs .vmd .citrix .com /XenServer /5.0.0/1.0/en _gb /reference .html #disk _qos
http ://docs .vmd .citrix .com /XenServer /5.0.0/1.0/en _gb /reference .html #disk _scheduler

linux kernel tuning for low latency
http ://quantlabs .net /blog /2011/09/quant -development -choosing -and -configuring -linux -for -low -latency /
http ://docs .redhat .com /docs /en -US /Red _Hat _Enterprise _MRG /2/html /Realtime _Tuning _Guide /chap -Realtime _Tuning _Guide -General _System _Tuning .html *************************************
http ://www .redhat .com /solutions /financial /trade _performance .html ****
http ://www .fizyka .umk .pl /~jkob /prace -mag /cfs -tuning .pdf ****************
http ://www .ibm .com /developerworks /linux /library /l -cfs / *******************
http ://www .softpanorama .org /Commercial _linuxes /performance _tuning .shtml ********
http ://lse .sourceforge .net /io /aio .html libaio *************
http ://www .redhat .com /f /pdf /summit /RedHatEnterprisePerfTuning .pdf
http ://doc .opensuse .org /products /opensuse /openSUSE /opensuse -tuning /cha .tuning .network .html ******
http ://www .psc .edu /networking /projects /tcptune *******************
http ://www .redbooks .ibm .com /redpapers /abstracts /redp 4285.html ************
http ://www .linuxvox .com /2009/11/what -is -the -linux -kernel -parameter -tcp _low _latency / ****
http ://hackingnasdaq .blogspot .com /2010/01/myth -of -procsysnetipv 4 tcplowlatency .html ************************

block devices
https ://computing .llnl .gov /linux /ucrl -id -144213.html ****
http ://dsstos .blogspot .com /2009/07/map -disk -block -devices -on -linux -host .html
ftp ://ftp .ddnsupport .com /pub /da 66 ba 4 d /DCS 9550_FC _LINUX _FAQ _v 1.6.pdf **
http ://www .puschitz .com /TuningLinuxForOracle .shtml #IOScheduler
http ://linux .web .cern .ch /linux /scientific 5/docs /rhel /Online _Storage _Reconfiguration _Guide .pdf
http ://www .monperrus .net /martin /scheduler +queue +size +and +resilience +to +heavy +IO *
http ://yoshinorimatsunobu .blogspot .com /2009/04/linux -io -scheduler -queue -size -and .html *
http ://talk .maemo .org /showthread .php ?t =69973 ***
http ://www .redhat .com /magazine /008 jun 05/features /schedulers / **
http ://www .softpanorama .org /Commercial _linuxes /Performance _tuning /disk _subsystem _tuning .shtml
http ://www .ibm .com /developerworks /wikis /display /LinuxP /Performance +Tuning
http ://docs .redhat .com /docs /en -US /Red _Hat _Enterprise _Linux /5/html /Oracle _Tuning _Guide /RHELTuningandOptimizationforOracleV 11.pdf***********
http ://www .redhat .com /f /pdf /summit /RedHatEnterprisePerfTuning .pdf *******
http ://www .redbooks .ibm .com /redpapers /pdfs /redp 4285.pdf *******
search on the web:

Whitepapers Red Hat Enterprise Linux 5 IO Tuning Guide Performance Tuning Whitepaper for Red Hat Enterprise Linux 5.2

http ://www .open -mag .com /features /Vol _98/QLogic /QLogic .htm **************
http ://filedownloads .qlogic .com /files /ms /74234/R 2_QLogic _Edits _SC _VMM 2008_PRO _Solution _Brief .pdf
http ://searchstorage .techtarget .com /tip /QLogics -2 GB -s -switches -missed -the -kudos -they -deserved
http ://www .softpanorama .org /Commercial _linuxes /Performance _tuning /troubleshooting _linux _performance _issues .shtml*******
http ://www .petroskoutoupis .com /lib /PetrosKoutoupis _EN 32009.pdf
http ://wikis .sun .com /download /attachments /216507788/03-ST 6000-Host -Attach -Linux -RalfWerner .pdf ***
http ://kernelnewbies .org /Linux _2_6_32#head -efc 263362 c 0 c 9 f 40 eb 83 bb 2 d 5 a 224057 ebf 7 a 59 f
http ://support .sas .com /resources /papers /proceedings 11/72480_RHEL 6_Tuning _Tips .pdf
http ://www .ufsdump .org /papers /io -tuning .pdf ****
http ://www .mjmwired .net /kernel /Documentation /block /deadline -iosched .txt **********
http ://kerneltrap .org /node /431 **********
http ://docs .redhat .com /docs /pt -BR /Red _Hat _Enterprise _Linux /index .html *
http ://support .sas .com /resources /papers /proceedings 10/FAQforStorageConfiguration .pdf ***
http ://support .sas .com /rnd /papers /sgf 07/sgf 2007-iosubsystem .pdf ***
http ://filedownloads .qlogic .com /files /driver /35491/README _qla 2 xxx 2-6.htm *******
http ://publib .boulder .ibm .com /infocenter /storwize /ic /index .jsp ?topic =%2 Fcom .ibm .storwize .v 7000.doc %2 Fsvc _linsetqdepth _1 dcv 4 w .html **
http ://virtualizationinformation .com /docs /PerformanceTuningVI 3 withNote .pdf ***
http ://www .redhat .com /f /pdf /RHEL 4_EQL _ORA 10 G _v 3.pdf ***** benchmarking comparação iSCSI x Fiber Channel
http ://phx .corporate -ir .net /phoenix .zhtml ?c =85695&p =irol -newsArticle &ID =211206&highlight = **** especificações da qla 2300.
http ://www .datasheetarchive .com /qlogic %202300-datasheet .html **** mais especificações da qla 2300
http ://www .nextre .it /oracledocs /ioscheduler _01.html ************
http ://talk .maemo .org /showthread .php ?t =70073 *******
http ://www .citrix .nl /site /resources /dynamic /partnerDocs /XSandNetAppstoragebestpractices _7.15.10.pdf Citrix XenServer and NetApp Storage Best Practices
http ://docs .vmd .citrix .com /XenServer /5.5.0/1.0/en _gb /reference .htm
http ://www .virtualistic .nl /archives /673 The complete Q&A from the Citrix XEN Masterclass webinar

Comentários