June 21st, 2010 | Tags: ,

In my new server, I have to do some fine tuning of MySQL 5 and here is what my.cnf file looks like for a double 2.33GHz CPU(four cores) machine with 16GB of memory.

Below are notes on some of the important variables, I took down while tuning the config file.

1. query_cache_size:

* MySQL provides one feature that can prove very handy – a query cache. In a situation where the database has to repeatedly run the same queries on the same data set, returning the same results each time, MySQL can cache the result set, avoiding the overhead of running through the data over and over and is extremely helpful on busy servers.

2. key_buffer_size:

* The value of key_buffer_size is the size of the buffer used with indexes. The larger the buffer, the faster the SQL command will finish and a result will be returned. The rule-of-thumb is to set the key_buffer_size to at least a quarter, but no more than half, of the total amount of memory on the server. Ideally, it will be large enough to contain all the indexes (the total size of all .MYI files on the server).
* A simple way to check the actual performance of the buffer is to examine four additional variables: key_read_requests, key_reads, key_write_requests, and key_writes.
* If you divide the value of key_read by the value of key_reads_requests, the result should be less than 0.01. Also, if you divide the value of key_write by the value of key_writes_requests, the result should be less than 1.

3. table_cache:

* The default is 64. Each time MySQL accesses a table, it places it in the cache. If the system accesses many tables, it is faster to have these in the cache. MySQL, being multi-threaded, may be running many queries on the table at one time, and each of these will open a table. Examine the value of open_tables at peak times. If you find it stays at the same value as your table_cache value, and then the number of opened_tables starts rapidly increasing, you should increase the table_cache if you have enough memory.

4. sort_buffer:

* The sort_buffer is very useful for speeding up myisamchk operations (which is why it is set much higher for that purpose in the default configuration files), but it can also be useful everyday when performing large numbers of sorts.

5. read_rnd_buffer_size:

* The read_rnd_buffer_size is used after a sort, when reading rows in sorted order. If you use many queries with ORDER BY, upping this can improve performance. Remember that, unlike key_buffer_size and table_cache, this buffer is allocated for each thread. This variable was renamed from record_rnd_buffer in MySQL 4.0.3. It defaults to the same size as the read_buffer_size. A rule-of-thumb is to allocate 1KB for each 1MB of memory on the server, for example 1MB on a machine with 1GB memory.

6. thread_cache:

* If you have a busy server that’s getting a lot of quick connections, set your thread cache high enough that the Threads_created value in SHOW STATUS stops increasing. This should take some of the load off of the CPU.

7. tmp_table_size:

* “Created_tmp_disk_tables” are the number of implicit temporary tables on disk created while executing statements and “created_tmp_tables” are memory-based. Obviously it is bad if you have to go to disk instead of memory all the time.

[mysqld]
socket=/path/to/mysql.sock
datadir=/var/lib/mysql
skip-locking
skip-innodb
# MySQL 4.x has query caching available.
# Enable it for vast improvement and it may be all you need to tweak.
query_cache_type=1
query_cache_limit=1M
query_cache_size=32M
# max_connections=500
# Reduced to 200 as memory will not be enough for 500 connections.
# memory=key_buffer+(sort_buffer_size+read_buffer_size)*max_connections
# which is now: 64 + (1 + 1) * 200 = 464 MB
# max_connections = approx. MaxClients setting in httpd.conf file
# Default set to 100.
#max_connections=200
#interactive_timeout=180
interactive_timeout=100
#wait_timeout=180
#wait_timeout=100
# Reduced wait_timeout to prevent idle clients holding connections.
#wait_timeout=30
wait_timeout=15
connect_timeout=10
# max_connect_errors is set to 10 by default
#max_connect_errors=10
#table_cache=256
#table_cache=1024
# Checked opened tables and adjusted accordingly after running for a while.
table_cache=512
#tmp_table_size=32M by default
#thread_cache=128
# Reduced it to 32 to prevent memory hogging. Also, see notes below.
thread_cache=32
# key_buffer=258M
# Reduced it by checking current size of *.MYI files, see notes below.
key_buffer=128M
# Commented out the buffer sizes and keeping the default.
# sort_buffer_size=2M by default.
#sort_buffer_size=1M
# read_buffer_size=128K by default.
#read_buffer_size=1M
# 1Mb of read_rnd_buffer_size for 1GB RAM — see notes below.
# read_rnd_buffer_size=256K by default.
#read_rnd_buffer_size=1M
# myisam_sort_buffer_size used for ALTER, OPTIMIZE, REPAIR TABLE commands.
# myisam_sort_buffer_size=8M by default.
#myisam_sort_buffer_size=64M
# thread_concurrency = 2 * (no. of CPU)
thread_concurrency=16
# log slow queries is a must. Many queries that take more than 2 seconds.
# If so, then your tables need enhancement.
log_slow_queries=/var/log/mysqld.slow.log
long_query_time=2

[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open_files_limit=8192

[mysqldump]
quick
max_allowed_packet=16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout

[client]
socket=/path/to/mysql.sock

June 19th, 2010 | Tags:

Emacs Makes A Computer Slow
Escape Meta Alt Control Shift
Emacs Makers Are Crazy Sickos
Emacs Makes All Computing Simple
Emacs Makefiles Annihilate C-Shells
Emacs Manuals Always Cause Senility
Emacs May Allow Customized Screwups
Emacs Manuals Are Cryptic and Surreal
Eventually Munches All Computer Storage
Eight Megabytes And Constantly Swapping
Elsewhere Maybe All Commands are Simple
Excellent Manuals Are Clearly Suppressed
Emacs May Alienate Clients and Supporters
Except by Middle Aged Computer Scientists
Extended Macros Are Considered Superfluous
Every Mode Accelerates Creation of Software
Each Manual’s Audience is Completely Stupefied
Exceptionally Mediocre Algorithm for Computer Scientists
Easily Maintained with the Assistance of Chemical Solutions
Eradication of Memory Accomplished with Complete Simplicity

April 8th, 2010 | Tags:

Unless you have a reason for doing otherwise, we recommend that you create the following partitions for x86 and AMD64 systems:

1. A swap partition (at least 256 MB) – swap partitions are used to support virtual memory. In other words, data is written to a swap partition when there is not enough RAM to store the data your system is processing. If you are unsure about what size swap partition to create, make it twice the amount of RAM on your machine, but no more than 2048 MB (or 2 GB). It must be of type swap.For example, if you have 1 GB of RAM or less, your swap partition should be a t least equal to the amount of RAM on your system, up to two times the RAM. For more than 1 GB of RAM , 2 GB of swap is recommended. Creating a large swap space partition can be especially hel pful if you plan to upgrade your RAM at a later time.

Tip: If your partitioning scheme requires a swap partition that is larger than 2 GB, you should create an additional swap partition. For example, if you have 4 GB of RAM, you may want to create two 2 GB swap partitions.

2. A /boot/ partition (100 MB) – the partition moun ted on /boot/ contains the operating system kernel (which allows your system to boot Red Hat Enterprise Linux, along with fi les used during the bootstrap process. Due to the limitations of most PC BIOSes, creating a smal l partition to hold these files is a good idea. For most users, a 100 MB boot partition is sufficient.

Caution: If your hard drive is more than 1024 cylinders (and your system was manufactured more than two years ago), you may need to create a /boot/ partition if you want the / (root) partition to use all of the remaining space on your hard drive.

Caution: If you have a RAID card, be aware that some BIOSes do not support booting from the RAID card. In cases such as these, the /boot/ partition must be created on a partition outside of the RAID array, such as on a separate hard drive.

3. A root partition (350 MB – 5.0 GB) – this is where “/ ” (the root directory) is located. In this setup, all files (except those stored in /boot ) are on the root partition. A 350 MB partition allows you to install a minimal installation, while a 5.0 GB root partition lets you perform a full installation, choosing all package groups.

4. A /var/ partition (3.0 GB or larger) – the /var/ partition is where variable data files are written. This includes spool directories and files, administrative and logging data, and transient and temporary files. Updates that are applied to Red Hat Enterprise Linux are written to the /var/ partition.

    February 23rd, 2010 | Tags:

    The ideal number of arguments for a function is zero (niladic).

    Next comes one (monadic), followed closely by two (dyadic).

    Three arguments (triadic) should be avoided where possible.

    More than three (polyadic) requires very special justification—and then shouldn’t be used anyway.

    February 10th, 2010 | Tags:

    Sharing knowledge is not about giving people something,
    or getting something from them. That is only valid for information sharing.
    Sharing knowledge occurs when people are genuinely interested in helping one another develop new capacities for action;
    it is about creating learning processes.

    February 10th, 2010 | Tags:

    http://www.merriam-webster.com/

    http://www.macmillandictionary.com/

    http://dictionary.cambridge.org/

    http://www.ldoceonline.com/

    http://www.collinslanguage.com/

    http://www.oup.com/elt/catalogue/teachersites/oald7/lookup?cc=global

    February 10th, 2010 | Tags:

    Recently, I saw someone that had developed a script on the CTP3 drop and was then having trouble running it on v1 of PowerShell.  Eventually it turned out that he was using v2 features in his script.  Most of you know that we are trying to keep the next version of PowerShell compatible with v1 and I encourage you to report any problems that you might have in that area.  Unfortunately, v2 being compatible with v1, doesn’t mean v1 is compatible with v2.

    There are many ways you could annotate your scripts so people know what version they are supposed to be run on, but PowerShell already has a way to do it.  It’s called #requires.

    In the majority of cases, all it will amount to is putting “#requires –version 2“  at the top of your script.  For example, here’s my awesome script that will only run on v2.

    #requires -version 2

    write-host “Can only run on v2.”

    And here are the results on v2, followed by v1:

    # V2

    PS>.\foo.ps1

    Can only run on v2.

    # V1

    PS > .\foo.ps1

    The script ‘foo.ps1′ cannot be run because it contained a “#requires” statement at line 1 for Windows PowerShell version 2.0 which is incompatible with the installed Windows PowerShell version of 1.0.

    At line:1 char:10

    + .\foo.ps1 <<<<

    PS >

    January 21st, 2010 | Tags:

    It is important to note that cmdlet parameters can have assigned alias names. They are not always easy to find but they are there. This is why you can say -ea instead of -errorAction. Use this code to locate the alias parameter names of any cmdlet:

    ‘Get-Childitem’ |
    Foreach-Object {
    (get-command $_).parameters |
    % { $_.Values |
    Where-Object { $_.Aliases.Count -gt 0 } |
    Select-Object Name, Aliases
    }
    }

    You can accomplish this by simply replacing the name of the cmdlet you want to examine at the start of the pipeline.

    October 12th, 2009 | Tags: ,

    There are two parameters we can play with:
    - the maximum number of allowed conntrack entries, which will be called
    CONNTRACK_MAX in this document
    - the size of the hash table storing the lists of conntrack entries, which
    will be called HASHSIZE (see below for a description of the structure)

    CONNTRACK_MAX is the maximum number of “sessions” (connection tracking entries)
    that can be handled simultaneously by netfilter in kernel memory.

    A conntrack entry is stored in a node of a linked list, and there are several
    lists, each list being an element in a hash table.  So each hash table entry
    (also called a bucket) contains a linked list of conntrack entries.
    To access a conntrack entry corresponding to a packet, the kernel has to:
    - compute a hash value according to some defined characteristics of the packet.
    This is a constant time operation.
    This hash value will then be used as an index in the hash table, where a
    list of conntrack entries is stored.
    - iterate over the linked list of conntrack entries to find the good one.
    This is a more costly operation, depending on the size of the list (and on
    the position of the wanted conntrack entry in the list).

    The hash table contains HASHSIZE linked lists.  When the limit is reached
    (the total number of conntrack entries being stored has reached CONNTRACK_MAX),
    each list will contain ideally (in the optimal case) about
    CONNTRACK_MAX/HASHSIZE entries.

    The hash table occupies a fixed amount of non-swappable kernel memory,
    whether you have any connections or not.  But the maximum number of conntrack
    entries determines how many conntrack entries can be stored (globally into the
    linked lists), i.e. how much kernel memory they will be able to occupy at most.

    This document will now give you hints about how to choose optimal values for
    HASHSIZE and CONNTRACK_MAX, in order to get the best out of the netfilter
    conntracking/NAT system.

    Default values of CONNTRACK_MAX and HASHSIZE
    ============================================

    By default, both CONNTRACK_MAX and HASHSIZE get average values for
    “reasonable” use, computed automatically according to the amount of
    available RAM.

    Default value of CONNTRACK_MAX
    ——————————

    On i386 architecture, CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 =
    RAMSIZE (in MegaBytes) * 64.
    So for example, a 32 bits PC with 512MB of RAM can handle 512*1024^2/16384 =
    512*64 = 32768 simultaneous netfilter connections by default.

    But the real formula is:
    CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (x / 32)
    where x is the number of bits in a pointer (for example, 32 or 64 bits)

    Please note that:
    - default CONNTRACK_MAX value will not be inferior to 128
    - for systems with more than 1GB of RAM, default CONNTRACK_MAX value is
    limited to 65536 (but can of course be set to more manually).

    Default value of HASHSIZE
    ————————-

    By default, CONNTRACK_MAX = HASHSIZE * 8.  This means that there is an average
    of 8 conntrack entries per linked list (in the optimal case, and when
    CONNTRACK_MAX is reached), each linked list being a hash table entry
    (a bucket).

    On i386 architecture, HASHSIZE = CONNTRACK_MAX / 8 =
    RAMSIZE (in bytes) / 131072 = RAMSIZE (in MegaBytes) * 8.
    So for example, a 32 bits PC with 512MB of RAM can store 512*1024^2/128/1024 =
    512*8 = 4096 buckets (linked lists)

    But the real formula is:
    HASHSIZE = CONNTRACK_MAX / 8 = RAMSIZE (in bytes) / 131072 / (x / 32)
    where x is the number of bits in a pointer (for example, 32 or 64 bits)

    Please note that:
    - default HASHSIZE value will not be inferior to 16
    - for systems with more than 1GB of RAM, default HASHSIZE value is limited
    to 8192 (but can of course be set to more manually).

    Reading CONNTRACK_MAX and HASHSIZE
    ==================================

    Current CONNTRACK_MAX value can be read at runtime, via the /proc filesystem.

    Before Linux kernel version 2.4.23, use:
    # cat /proc/sys/net/ipv4/ip_conntrack_max

    Since Linux kernel version 2.4.23 (thus Linux 2.6 as well), use:
    # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
    (old /proc/sys/net/ipv4/ip_conntrack_max is then deprecated!)

    Current HASHSIZE is always available (for every kernel version) in syslog
    messages, as the number of buckets (which is HASHSIZE) is printed there at
    ip_conntrack initialization.
    Since Linux kernel version 2.4.24 (thus Linux 2.6 as well), current HASHSIZE
    value can be read at runtime with:
    # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets

    Modifying CONNTRACK_MAX and HASHSIZE
    ====================================

    Default CONNTRACK_MAX and HASHSIZE values are reasonable for a typical host,
    but you may increase them on high-loaded firewalling-only systems.
    So CONNTRACK_MAX and HASHSIZE values can be changed manually if needed.

    While accessing a bucket is a constant time operation (hence the interest
    of having a hash of lists), keep in mind that the kernel has to iterate over
    a linked list to find a conntrack entry.  So the average size of a linked
    list (CONNTRACK_MAX/HASHSIZE in the optimal case when the limit is reached)
    must not be too big.  This ratio is set to 8 by default (when values are
    computed automatically).
    On systems with enough memory and where performance really matters, you can
    consider trying to get an average of one conntrack entry per hash bucket,
    which means HASHSIZE = CONNTRACK_MAX.

    Setting CONNTRACK_MAX
    ———————

    Conntrack entries are stored in linked lists, so the maximum number of
    conntrack entries (CONNTRACK_MAX) can be easily configured dynamically.

    Before Linux kernel version 2.4.23, use:
    # echo $CONNTRACK_MAX > /proc/sys/net/ipv4/ip_conntrack_max

    Since Linux kernel version 2.4.23 (thus Linux 2.6 as well), use:
    # echo $CONNTRACK_MAX > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

    where $CONNTRACK_MAX is an integer.

    Setting HASHSIZE
    —————-

    For mathematical reasons, hash tables have static sizes.  So HASHSIZE must be
    determined before the hash table is created and begins to be filled.

    Before Linux kernel version 2.4.21, a prime number should be chosen for hash
    size, ensuring that the hash table will be efficiently populated. Odd
    non-prime numbers or even numbers are strongly discouraged, as the hash
    distribution will be sub-optimal.

    Since Linux kernel version 2.4.21 (thus Linux 2.6 as well), conntrack
    uses jenkins2b hash algorithm which is happy with all sizes, but power
    of 2 works best.

    If netfilter conntrack is statically compiled in the kernel, the hash table
    size can be set at compile time, or (since kernel 2.6) as a boot option with
    ip_conntrack.hashsize=$HASHSIZE

    If netfilter conntrack is compiled as a module, the hash table size can be
    set at module insertion, with the following command:
    # modprobe ip_conntrack hashsize=$HASHSIZE

    where $HASHSIZE is an integer.

    Since 2.6.14, it is possible to set hashsize dynamically at runtime,
    after boot and module load.

    Between 2.6.14 and 2.6.19 (included), use:
    # echo $HASHSIZE > /sys/module/ip_conntrack/parameters/hashsize

    Since 2.6.20, use:
    # echo $HASHSIZE > /sys/module/nf_conntrack/parameters/hashsize

    Ideal case: firewalling-only machine
    ————————————

    In the ideal case, you have a machine _just_ doing packet filtering and NAT
    (i.e. almost no userspace running, at least none that would have a growing
    memory consumption like proxies, …).

    The size of kernel memory used by netfilter connection tracking is:
    size_of_mem_used_by_conntrack (in bytes) =
    CONNTRACK_MAX * sizeof(struct ip_conntrack) +
    HASHSIZE * sizeof(struct list_head)
    where:
    - sizeof(struct ip_conntrack) can vary quite much, depending on architecture,
    kernel version and compile-time configuration. To know its size, see the
    kernel log message at ip_conntrack initialization time.
    sizeof(struct ip_conntrack) is around 300 bytes on i386 for 2.6.5, but
    heavy development around 2.6.10 make it vary between 352 and 192 bytes!
    - sizeof(struct list_head) = 2 * size_of_a_pointer
    On i386, size_of_a_pointer is 4 bytes.

    So, on i386, kernel 2.6.5, size_of_mem_used_by_conntrack is around
    CONNTRACK_MAX * 300 + HASHSIZE * 8 (bytes).

    If we take HASHSIZE = CONNTRACK_MAX (if we have most of the memory dedicated
    to firewalling, see “Modifying CONNTRACK_MAX and HASHSIZE” section above),
    size_of_mem_used_by_conntrack would be around CONNTRACK_MAX * 308 bytes
    on i386 systems, kernel 2.6.5.

    Now suppose your firewalling-only box has 512MB of RAM (a decent amount
    of memory considering today’s memory prices). You have to spare a bit of
    memory for a few applications (syslog, etc.): 128MB should really be big
    enough for a firewall in console mode, for example.
    The rest can be dedicated to conntrack entries.
    Then you could set both CONNTRACK_MAX and HASHSIZE approximately to:
    (512 – 128) * 1024^2 / 308 =~ 1307315 (instead of 32768 for CONNTRACK_MAX,
    and 4096 for HASHSIZE by default).
    Since Linux 2.4.21 (thus Linux 2.6 as well), hash algorithm is happy with
    “power of 2″ sizes (it used to be a prime number before).

    So here we can set CONNTRACK_MAX and HASHSIZE to 1048576 (2^20), for example.

    This way, you can store about 32 times more conntrack entries than the
    default, and get better performance for conntrack entry access.

    Latest version of this document can be found at:
    http://www.wallfire.org/misc/netfilter_conntrack_perf.txt

    October 4th, 2009 | Tags: , ,

    If you are a developer you need C and C++ Compiler for your development work. In ubuntu you can install the build-essential for C and C++ compilers.

    Install C and C++ Compilers in Ubuntu

    sudo aptitude install build-essential

    This will install all the required packages for C and C++ compilers

    Testing C and C++ Programs

    1. Compiling Your first C Programs

    Now you need to open first.c file

    gedit first.c

    add the following lines save and exit the file

    #include <stdio.h>
    int main() {
    printf(“Hello, world\n”);
    return 0;
    }

    Firstly compile the code using the following command

    cc -c first.c

    that would produce an object file you may need to add to the library.

    then create an executable using the following command

    cc -o first first.c

    Now run this executable using the following command

    ./first

    Output should show as follows

    Hello, world

    2. Compiling your first C++ program

    If you want to run c++ program follow this procedure

    g++ is the compiler that you must use.

    you should use a .cpp file extension rather than a .c one

    You need to create a file

    gedit first.cpp

    add the following lines save and exit the file

    #include <iostream>
    int main() {
    std::cout << “Hello World!” << std::endl;
    return 0;
    }

    Run your C++ Program using the following command

    g++ first.cpp -o test

    ./test

    Output should show as follows

    Hello World!