Novos blogs:  postgresql extreme tuning 40 cores 512 GB RAM Debian  debian linux kernel tuning high io  ZFS parte 6: vídeo palestra ZFS para gestores e sysadmins  ZFS parte 5: como comprar ou montar ZFS data storage server  PostgreSQL tuning example


Tech Force / Linux blog / Debian Linux kernel tuning low latency small files



Right menu

Linux blog recente

PostgreSQL: palestra tuning extremo em hardware 40 núcleos, 80 threads, 512 GB RAM, Debian

Palestra com dicas de tuning extremo para dezenas de milhares de TPS e IOPS com PostgreSQL sobre Debian GNU/Linux.

Não conectado

Notificação


Debian GNU / Linux kernel tuning for low latency on small files

E-mail servers, data base servers, file servers, web servers struggle to cope with LATENCY on accessing millions of small files under heavy concurrent load.

Low latency is key factor to a good user experience.

With the Debian GNU / Linux 6.x kernel tuning, combined with the previous article hints about XenServer I/O latency, tuning filesystems configuring multipath, and data storage, we reduced eighty-four times our virtual block device latency at our environment.

Thousands concurrent synchronous writes of random small files on filesystems with millions of them while honoring write barriers for POSIX compliance, stress the limits of the data storage, file systems, kernel I/O, virtualization I/O, hardware data paths.

These are our findings and Linux kernel tunings in Cyrus IMAP e-mail servers grid under heavy load production since November 2011.

First of all, do not blindly apply these settings at your servers. Read the whole bibliography to understand what you are about to tune.

Understand what each parameter is doing.

Understand your server workload profile pattern.

Understand your hardware and storage performance and behaviour.

Understand your Fiber Channel SAN or iSCSI data path and network segment behaviour.

OUR environment has servers using Fiber Channel HBA multipath connected to FC disks WAFL Data Storage with very low latency and lots of ECC NVRAM for cache. It almost behaves as a giant SSD storage from latency standpoint.

This is important: if you are going to “flush” your servers kernel buffers as fast they could, you data storage and data path should be able to cope with such high IOPS and trough output.

You could even get out-of-memory errors caused by high latency storage hardware. The linked article author gone the opposite direction in tuning due to HIS hardware environment.

Read YOUR deployed kernel version documentation. The links below are for the latest kernel.org version.

http://www.kernel.org/doc/Documentation/block/queue-sysfs.txt

http://www.kernel.org/doc/Documentation/block/deadline-iosched.txt

http://www.kernel.org/doc/Documentation/sysctl/vm.txt

http://www.nextre.it/oracledocs/ioscheduler_01.html

Also, the net result is only as fast as your slowest link at your application stack, hardware and network stacks.

Carefully configure your application stack for low latency and performance.

At other articles we covered filesystem tuning, data storage LUN configurations, and some aspects of Cyrus IMAP configurations. More on future articles about performance tuning.

You will carefully watch the iostat output during the tests, looking for the bottlenecks.

Read the previous article bibliography and test results about XenServer I/O latency.

Learn how to correlate the output columns with the kernel parameters and your data storage behaviour. Read the manpage of iostat.

iostat -dmxthN 1 /dev/xvd*

Non persistent between reboots configuration

Use these during your test, tuning and evaluation phase.

Logged as root:

sysctl vm.swappiness=10 

sysctl vm.dirty_background_ratio=1 

sysctl vm.dirty_expire_centisecs=500 

sysctl vm.dirty_ratio=15 

sysctl vm.dirty_writeback_centisecs=100 


cat /sys/block/xvdb/queue/nr_requests 

echo "4" > /sys/block/xvdb/queue/nr_requests 

cat /sys/block/xvdb/queue/nr_requests 

cat /sys/block/xvdb/queue/scheduler 

echo deadline > /sys/block/xvdb/queue/scheduler 

cat /sys/block/xvdb/queue/scheduler 


cat /sys/block/xvdb/queue/iosched/front_merges 

echo 1 >/sys/block/xvdb/queue/iosched/front_merges

cat /sys/block/xvdb/queue/iosched/front_merges 

cat /sys/block/xvdb/queue/iosched/fifo_batch 

echo 1 >/sys/block/xvdb/queue/iosched/fifo_batch 

cat /sys/block/xvdb/queue/iosched/fifo_batch

A very aggressive configuration (watch your %sys during test phase):

sysctl vm.dirty_expire_centisecs=50 

sysctl vm.dirty_writeback_centisecs=10 

sysctl vm.dirty_background_ratio=0

Debian GNU/Linux configuration persistent between reboots

You will need the “sysfsutils” package installed.

Append to your /etc/sysfs.conf

#AFM 20120523

block/xvdb/queue/nr_requests = 4 

block/xvdb/queue/scheduler = deadline 

block/xvdb/queue/iosched/front_merges = 1 

block/xvdb/queue/iosched/fifo_batch = 1

Append to your /etc/sysctl.conf

#AFM 20120523 

vm.swappiness = 10 

vm.dirty_background_ratio = 1 

vm.dirty_expire_centisecs = 500 

vm.dirty_ratio = 15 

vm.dirty_writeback_centisecs = 100

Some default values for the deadline i/o scheduler

For Debian GNU / Linux 6.x Squeeze, kernel 2.6.32

cat /sys/block/xvdb/queue/iosched/fifo_batch
16
cat /sys/block/xvdb/queue/iosched/front_merges
1 
cat /sys/block/xvdb/queue/iosched/read_expire
500 
cat /sys/block/xvdb/queue/iosched/write_expire
5000 
cat /sys/block/xvdb/queue/iosched/writes_starved
2

Bibliography

kswapd0 , ksoftirqd0

http://forums.citrix.com/message.jspa?messageID=1511105 ************************** tuning de kernel para arquivos pequenos

http://askubuntu.com/questions/7858/why-is-ksoftirqd-0-process-using-all-my-cpu *

http://askubuntu.com/questions/73639/what-are-kswapd0-kworker-numnum-ksoftirqd-num *

http://www.mail-archive.com/rhelv5-list@redhat.com/msg05345.html

http://lkml.indiana.edu/hypermail/linux/kernel/0506.0/1150.html

http://grokbase.com/t/centos.org/centos/2011/05/centos-kswapd-taking-100-cpu-with-no-swap-on-system/07g622yntfakij4lsruvrqxyjeom

http://answers.softpicks.net/answers/topic/High-CPU-load-with-kswapd-and-heavy-disk-I-O-1747240-1.htm

https://bugzilla.redhat.com/show_bug.cgi?id=115438

http://www.spinics.net/lists/linux-mm/msg16292.html

swappiness

http://feedblog.org/2006/09/27/stupid-linux-swap-tricks-with-swappiness/

http://www.linuxvox.com/2009/10/what-is-the-linux-kernel-parameter-vm-swappiness/

https://help.ubuntu.com/community/SwapFaq

http://kerneltrap.org/node/3000

http://lists.centos.org/pipermail/centos/2011-May/111433.html

https://bugzilla.redhat.com/show_bug.cgi?id=135312

https://bugs.launchpad.net/ubuntu/+bug/721896

https://bugzilla.kernel.org/show_bug.cgi?id=12309

http://us.generation-nt.com/kswapd-causing-giant-load-help-179939921.html

http://www.linuxquestions.org/questions/linux-server-73/100-swap-usage-with-loads-of-free-memory-849452/

https://www.centos.org/modules/newbb/viewtopic.php?topic_id=28555

https://bugzilla.redhat.com/show_bug.cgi?id=437202

http://kerneltrap.org/mailarchive/linux-kernel/2007/9/24/294490

http://www.linuxforums.org/forum/kernel/65380-what-does-kswapd0-do.html

Linux kernel tuning for low latency block devices and small files

http://www.cyrius.com/debian/nslu2/linux-on-flash.html

http://www.kernel.org/doc/Documentation/sysctl/vm.txt ****************

http://serverfault.com/questions/126413/limit-linux-background-flush-dirty-pages ******

http://www.westnet.com/~gsmith/content/linux-pdflush.htm ****************

http://www.redhat.com/archives/linux-lvm/2004-February/msg00115.html ****

http://www.fccps.cz/download/adv/frr/hdd/hdd.html#tuning-intro ***************

http://www.serverphorums.com/read.php?12,115991

http://www.r71.nl/kb/technical/102-tuning-journaling-file-systems

http://support.citrix.com/article/CTX127065

http://forums.citrix.com/thread.jspa?messageID=1487300

http://knol.google.com/k/linux-performance-tuning-and-measurement *********

http://kerneltrap.org/node/4462 ***

http://www.mjmwired.net/kernel/Documentation/block/barrier.txt *****

http://en.wikipedia.org/wiki/Slab_allocation *

http://www.ibm.com/developerworks/linux/library/l-linux-slab-allocator/

http://www.win.tue.nl/~aeb/linux/lk/lk-9.html **

http://www.fccps.cz/download/adv/frr/hdd/hdd.html ****

http://makarevitch.org/rant/raid/

http://www.softpanorama.org/Admin/performance_monitoring.shtml

http://www.linuxinsight.com/proc_sys_vm_vfs_cache_pressure.html *******

http://rudd-o.com/en/linux-and-free-software/tales-from-responsivenessland-why-linux-feels-slow-and-how-to-fix-that

https://bbs.archlinux.org/viewtopic.php?id=40111

http://duopetalflower.blogspot.com/2009/11/tuning-ubuntu-910-karmic-for-speed-and.html

http://www.pythian.com/news/247/basic-io-monitoring-on-linux/ ***

http://www.informit.com/articles/article.aspx?p=481867 ***

http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/ ****

http://www.xaprb.com/blog/2009/08/23/how-to-find-per-process-io-statistics-on-linux/

http://www.faqs.org/docs/linux_admin/buffer-cache.html ***********

http://docs.vmd.citrix.com/XenServer/5.0.0/1.0/en_gb/reference.html#disk_qos

http://docs.vmd.citrix.com/XenServer/5.0.0/1.0/en_gb/reference.html#disk_scheduler

linux kernel tuning for low latency

http://quantlabs.net/blog/2011/09/quant-development-choosing-and-configuring-linux-for-low-latency/

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/2/html/Realtime_Tuning_Guide/chap-Realtime_Tuning_Guide-General_System_Tuning.html *************************************

http://www.redhat.com/solutions/financial/trade_performance.html ****

http://www.fizyka.umk.pl/~jkob/prace-mag/cfs-tuning.pdf ****************

http://www.ibm.com/developerworks/linux/library/l-cfs/ *******************

http://www.softpanorama.org/Commercial_linuxes/performance_tuning.shtml ********

http://lse.sourceforge.net/io/aio.html libaio *************

http://www.redhat.com/f/pdf/summit/RedHatEnterprisePerfTuning.pdf

http://doc.opensuse.org/products/opensuse/openSUSE/opensuse-tuning/cha.tuning.network.html ******

http://www.psc.edu/networking/projects/tcptune *******************

http://www.redbooks.ibm.com/redpapers/abstracts/redp4285.html ************

http://www.linuxvox.com/2009/11/what-is-the-linux-kernel-parameter-tcp_low_latency/ ****

http://hackingnasdaq.blogspot.com/2010/01/myth-of-procsysnetipv4tcplowlatency.html ************************

block devices

https://computing.llnl.gov/linux/ucrl-id-144213.html ****

http://dsstos.blogspot.com/2009/07/map-disk-block-devices-on-linux-host.html

ftp://ftp.ddnsupport.com/pub/da66ba4d/DCS9550_FC_LINUX_FAQ_v1.6.pdf **

http://www.puschitz.com/TuningLinuxForOracle.shtml#IOScheduler

http://linux.web.cern.ch/linux/scientific5/docs/rhel/Online_Storage_Reconfiguration_Guide.pdf

http://www.monperrus.net/martin/scheduler+queue+size+and+resilience+to+heavy+IO *

http://yoshinorimatsunobu.blogspot.com/2009/04/linux-io-scheduler-queue-size-and.html *

http://talk.maemo.org/showthread.php?t=69973 ***

http://www.redhat.com/magazine/008jun05/features/schedulers/ **

http://www.softpanorama.org/Commercial_linuxes/Performance_tuning/disk_subsystem_tuning.shtml

http://www.ibm.com/developerworks/wikis/display/LinuxP/Performance+Tuning

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Oracle_Tuning_Guide/RHELTuningandOptimizationforOracleV11.pdf ***********

http://www.redhat.com/f/pdf/summit/RedHatEnterprisePerfTuning.pdf *******

http://www.redbooks.ibm.com/redpapers/pdfs/redp4285.pdf *******

search on the web:

  • Whitepapers Red Hat Enterprise Linux 5 IO Tuning Guide Performance Tuning Whitepaper for Red Hat Enterprise Linux 5.2

http://www.open-mag.com/features/Vol_98/QLogic/QLogic.htm **************

http://filedownloads.qlogic.com/files/ms/74234/R2_QLogic_Edits_SC_VMM2008_PRO_Solution_Brief.pdf

http://searchstorage.techtarget.com/tip/QLogics-2GB-s-switches-missed-the-kudos-they-deserved

http://www.softpanorama.org/Commercial_linuxes/Performance_tuning/troubleshooting_linux_performance_issues.shtml *******

http://www.petroskoutoupis.com/lib/PetrosKoutoupis_EN32009.pdf

http://wikis.sun.com/download/attachments/216507788/03-ST6000-Host-Attach-Linux-RalfWerner.pdf ***

http://kernelnewbies.org/Linux_2_6_32#head-efc263362c0c9f40eb83bb2d5a224057ebf7a59f

http://support.sas.com/resources/papers/proceedings11/72480_RHEL6_Tuning_Tips.pdf

http://www.ufsdump.org/papers/io-tuning.pdf ****

http://www.mjmwired.net/kernel/Documentation/block/deadline-iosched.txt **********

http://kerneltrap.org/node/431 **********

http://docs.redhat.com/docs/pt-BR/Red_Hat_Enterprise_Linux/index.html *

http://support.sas.com/resources/papers/proceedings10/FAQforStorageConfiguration.pdf ***

http://support.sas.com/rnd/papers/sgf07/sgf2007-iosubsystem.pdf ***

http://filedownloads.qlogic.com/files/driver/35491/README_qla2xxx2-6.htm *******

http://publib.boulder.ibm.com/infocenter/storwize/ic/index.jsp?topic=%2Fcom.ibm.storwize.v7000.doc%2Fsvc_linsetqdepth_1dcv4w.html **

http://virtualizationinformation.com/docs/PerformanceTuningVI3withNote.pdf ***

http://www.redhat.com/f/pdf/RHEL4_EQL_ORA10G_v3.pdf ***** benchmarking comparação iSCSI x Fiber Channel

http://phx.corporate-ir.net/phoenix.zhtml?c=85695&p=irol-newsArticle&ID=211206&highlight= **** especificações da qla 2300.

http://www.datasheetarchive.com/qlogic%202300-datasheet.html **** mais especificações da qla 2300

http://www.nextre.it/oracledocs/ioscheduler_01.html ************

http://talk.maemo.org/showthread.php?t=70073 *******

http://www.citrix.nl/site/resources/dynamic/partnerDocs/XSandNetAppstoragebestpractices_7.15.10.pdf Citrix XenServer and NetApp Storage Best Practices

http://docs.vmd.citrix.com/XenServer/5.5.0/1.0/en_gb/reference.htm

http://www.virtualistic.nl/archives/673 The complete Q&A from the Citrix XEN Masterclass webinar

How to configure maximum performance storage space for Debian GNU / Linux on IBM DS 8300 Data Storage Systems

The IBM DS 8300 Data Storage Systems are multi millions dollars flexible high availability and performance SAN machines.

But you may left much of such performance and availability behind if you do not configure them correctly for Debian GNU / Linux.

See how to ask for performance data storage space on them. Or what you need to configure on them.

Read about an actual configuration running with Debian GNU / Linux hosts at SERPRO.

How to configure multipath for high availability and performance on Debian and CentOS for storage at IBM DS8300 SAN

This detailed how to guides to achieve high availability and performance on Debian and CentOS for accessing storage space at IBM DS8300 Data Storage Systems.

Tested on Debian GNU/Linux 5.x Lenny 64 bits and CentOS 5.3 64 bits running on 8 cores blades, with Host Bus Adapters Qlogic and Emulex Light Pulse Fiber Channel in deployed systems at SERPRO.

Observations showed that Debian Lenny has the best performance, for our app load profile and hardware.

LVM, RAID, XFS and EXT3 file systems tuning for small files massive heavy load concurrent parallel I/O on Debian

Thousands concurrent parallel read write accesses over tens of millions of small files is a terrible performance tuning problem for e-mail servers.

You must understand and fine tune all your infrastructure chain, following the previous articles for data storage and multipath on Debian 5.x Lenny.

We reduced the CPU I/O wait from 30% to 0,3% (XFS) and 5% (EXT3) with these combined previously undocumented file system tuning tips.

XenServer: Como reduzir latência i/o rede e disco e reduzir perda de pacotes em alta carga

Para melhor experiência de usuário, devido a respostas mais rápidas do serviço, é fator chave reduzir a LATÊNCIA de I/O de rede e disco.

Especialmente em servidores de e-mail e de banco de dados.

Veremos como reduzir a latência de de I/O Máquinas Virtuais em servidores de virtualização XenServer.

Tutorial Cyrus IMAP aggregator (murder) 2.3.16 sobre Debian GNU Linux 5.x Lenny

Este tutorial completo mostrará como montar um LABORATÓRIO de infra-estrutura de e-mail bastante escalável, para centenas de milhares de contas.

Comentários

Usuários registrados têm permissão para criar comentários.


Translate this page.  

Add to Free Software Daily   
  
Follow meon