Use cases for FT ^
- An application that needs to be bulletproof: Its workload cannot have five minutes of downtime. It’s a business-critical enterprise application. Every minute counts.
- A custom application that cannot do clustering any other way.
- Temporary protection: You might have a monthly report running one day per month every month, and you want to be absolutely sure your VM won’t be down because of a hardware problem. You activate FT for a day for that particular VM.
- Clustering is too complicated to maintain and configure: There might be cases where your application can be clustered, but it’s really problematic and difficult to install and maintain.
Turning Fault Tolerance On
There are a few other requirements that must be met before you can enable FT on a VM. You need to ensure that the VM resides on shared storage (iSCSI, NFS,etc) and that is has no unsupported devices; see list above. Also, note that the disk provisioning type changes to Thick Provision Eager Zeroed after FT is turned on for a VM with thin disks. The conversion process may take a while for large vmdks.
FT can be turned on irrespective of the VM’s power state. However, a number of validation checks are carried out that might differ depending on whether the VM is on or not.
To turn on FT, just right-click on the VM and select Turn On Fault Tolerance. Use the same procedure to turn off FT.
Figure 4 – Turning on FT on a virtual machine
Figure 5 – Changes affected on a vm once FT is turned on
The next video, shows how to turn FT on and off. I’m using a 2-node cluster sporting nested hypervisors so performance is what it is.
https://youtube.com/watch?v=V3QBas1fW2c
Win2008-C
Figure 6 – vmdk scrubbing – changing provisioning type to thick eager zeroed
Figure 7 – Primary VM’s host
Figure 8 – Secondary VM’s host
Next, select the VM for which you turned on FT. Under the Summary tab, you should now find a Fault Tolerance details window giving you a number of FT metrics for the selected VM (Fig. 9). Of particular interest is the vLockstep interval which is a measure of how far behind the secondary VM is from the primary one in terms of changes that need to be replicated. Typically, the value should be less than 500ms. In the example shown below, the interval value is of 0.023s or 23ms which is within the acceptable range.
Another metric Log Bandwidth, gives you the current network bandwidth used to replicate changes between the primary and secondary VMs. This can quickly spike up when you have multiple FT protected VMs ((max. 4 VMs or 8 vCPUs per host) hence the 10Gbit dedicated network requirement for FT logging.
Figure 9 – Fault Tolerance details for a FT protect virtual machine
You can simulate a host failure by selecting Test Failover from the VM’s context menu. You can also simulate a secondary restart using the same menu. In both instances, the VM is unprotected while the test is in progress.
Figure 10 – Testing failover
In the next video, I show how to perform a failover and a secondary restart along with the following:
- I illustrate where from HA (and DRS) is enabled which in turn lets you deploy FT protected VMs.
- Verify that the primary and secondary VMs are hosted on different servers.
- Initiate a failover test while pinging an FT protected VM’s IP address. Normally, you’d see a single packet loss. The 3-packet loss you see, is due to the low performing nested environment used.
- Simulate a secondary restart. Notice that the VM is unprotected for a brief instance but returns to being fully protected after the simulation completes.
https://youtube.com/watch?v=TyK0t3OKxls
Совместимость гостевых ОС с технологией VMware Fault Tolerance.
Filed under:Application vServices, Fault Tolerance от Александр Самойленко
В продукте VMware vSphere 4 есть технология Fault Tolerance, позволяющая добиться непрерывной доступности для виртуальных машин и сервисов в гостевых ОС при отказе физических хост-серверов VMware ESX / ESXi. Но все ли гостевые ОС поддерживаются технологией VMware Fault Tolerance?
Ответ таков — технология VMware FT требует специфических процессоров и работает для определенных гостевых ОС виртуальных машин.
Для начала поддерживаемые процессоры:
Intel Xeon based on 45nm Core 2 Microarchitecture Category:
- 3100 Series
- 3300 Series
- 5200 Series (DP)
- 5400 Series
- 7400 Series
Intel Xeon based on Core i7 Microarchitecture Category:
5500 Series
AMD 3rd Generation Opteron Category:
- 1300 Series
- 2300 Series (DP)
- 8300 Series (MP)
Теперь, вот как поддерживаются гостевые ОС виртуальных машин при использовании технологии VMware Fault Tolerance:
| uest Operating System |
withIntel Xeon Based on 45nm Core 2 Microarchitecture Fault Tolerance Support |
Fault Tolerance Support with Intel Xeon Based on Core i7 Microarchitecture |
Fault Tolerance Support with AMD 3rd Generation Opteron |
| Windows Server 2008 | Yes | Yes/Off | Yes/Off |
| Windows Vista | Yes | Yes/Off | Yes/Off |
| Windows Server 2003 (64 bit) | Yes | Yes/Off | Yes/Off |
| Windows Server 2003 (32 bit) | Yes | Yes/Off |
Yes/Off (Requires Service Pack 2 or greater) |
| Windows XP (64 bit) | Yes | Yes/Off | Yes/Off |
| Windows XP (32 bit) | Yes | Yes/Off | No |
| Windows 2000 | Yes/Off | Yes/Off | No |
| Windows NT 4.0 | Yes/Off | Yes/Off | No |
| Linux (all ESX-supported distributions) | Yes | Yes/Off | Yes/Off |
| Netware Server | Yes/Off | Yes/Off | Yes/Off |
| Solaris 10 (64-bit) | Yes | Yes/Off |
Yes/Off (Requires Solaris U1) |
| Solaris 10 (32-bit) | Yes | Yes/Off | No |
| FreeBSD (all ESX-supported distributions) | Yes | Yes/Off | Yes/Off |
В таблице значение “Yes” означает, что технология поддерживается,
значение “Off” — для включения Fault Tolerance требуется выключение виртуальной машины,
а значение “No” означает отсутствие поддержки FT для данной гостевой ОС.
Для проверки совместимости хостов VMware ESX / ESXi с технологией Fault Tolerance можно использовать утилиту SiteSurvey от VMware.
How vSphere Fault Tolerance works ^
To put it simply, vSphere FT works by continuously replicating an entire running VM from one physical server to another. The FT-enabled VM has two replicas:
- Primary VM
- Secondary VM
Each VM is running on a different ESXi host. The replicas are logically identical; they represent a single VM state and a single network identity, but they are physically distinct.
vSphere 6.5 FT protected VM
Each replica has its own virtual machine files, such as configuration files (VMX) and virtual machine disk files (VMDK).
After activation of FT, the first synchronization of the virtual machine disk files (VMDKs) happens using vSphere Storage vMotion. Subsequently, vSphere FT will mirror VMDK writes between the primary and secondary VM over the FT network.
To check the VM’s state, you can also see the dashboard widget with detailed log bandwidth usage.
Virtual machine running with FT protection
When a physical server fails, VMware HA automatically restores redundancy by restarting a new secondary VM on another host. The VM state, network identity, and all active network connections for the VM will be identical, reflecting the whole state as a primary VM again. If the host running the secondary VM fails, VMware HA starts a new secondary VM on a different host.
Another improvement of VMware FT is that you can now configure FT networks to use multiple network interface controllers (NICs) to increase the overall bandwidth for FT logging traffic. This works similarly to Multi-NIC vMotion and provides more bandwidth for the FT network.
Сигналы
В UNIX-подобных системах сигналы – метод уведомления процессов о событиях. Они выполняются асинхронно и с высоким приоритетом, так что при получении такого notification система прерывает выполнение процесса.
При чем здесь сигналы в статье о fault tolerance? Функция подходит для остановки или перезапуска процессов (демонов) в случае ошибки, и корректного завершения задачи.
PCNTL
Для PHP доступно расширение-менеджер процессов pcntl. Оно использует UNIX-стиль управления процессами и подпрограммами, в том числе обработку сигналов. Так что на его основе реализуется простейшая отказоустойчивая система:
class SIG { public static $cought = 0; }
…
function sig_handler($sig) {
SIG::$cought = $sig;
echo ‘Finishing all operations to exit…’ . “n”;
}
pcntl_signal(SIGINT, “sig_handler”);
pcntl_signal(SIGTERM, “sig_handler”);
pcntl_signal(SIGHUP, “sig_handler”);
while ( true )
{
/*
Длинный код скрипта для обработки с нужными действиями
*/
…
if ( SIG::$cought )
{
echo ‘finished operations, exiting now’ . “n”;
exit;
}
}
## Отлавливает сигналы SIGINT, SIGTERM, SIGHUP, если такой сигнал приходит, то скрипт останавливается
Такой подход используется для корректного перезапуска скриптов при использовании supervisor.
pcntl подходит для асинхронного выполнения PHP как альтернатива pthreads. Отличия между расширениями в том, что pcntl – менеджер процессов, а pthreads – менеджер потоков.
Best practices for VMware FT ^
You should follow some best practices when thinking of using FT within your virtual infrastructure.
- You should use approximately the same CPU frequencies on the hosts where the primary and secondary VMs will run. This is because the secondary VM might restart more frequently; configuring the host enforces low-frequency mode to save power. (Check the BIOS setup and configure your host not to use the economy energy mode for the CPU).
- You should configure your BIOS across all hosts to apply the same instruction set (enabled or disabled). You should also check your motherboard documentation on how to enable or disable instruction sets on your systems.
And you should also use:
- Processors from the same compatible processor group
- The same ESXi version
- Common access to data stores the VMs use
- The same VM network configuration
Where do you activate VMware Fault Tolerance? ^
First, on each host, create a new VMkernel network adapter and activate Fault Tolerance logging.
Start the vSphere Web Client and go to Host > Manage > Networking. Then go to VMkernel adapters > Add host networking; select the VMkernel network adapter and select an existing standard switch or a new vSwitch. On the Port properties, enable Fault Tolerance.
Configure the VMkernel network adapter for Fault Tolerance logging
In addition, you must enable the vMotion VMkernel interface. Also, make sure the host has a license covering Fault Tolerance. Then, at the VM level, you can configure VMware FT.
It’s fairly simple. Once you’ve met the requirements, simply right-click a VM and go to Fault Tolerance > Turn on Fault Tolerance.
Turn On Fault Tolerance
Типовые ошибки VMWare Fault Tolerance
1. Отсутствует FT VMkernel
Причина:
Для работы технологии FT требуется специальный сетевой интерфейс, который будет использоваться для копирования данных и логов с Primary виртуальной машины на Secondary. Для этих целей необходимо создать отдельный порт VMkernel, либо же задействовать существующий. В моем, случае, для этих целей я задействую выделенную сеть для vMotion, т.к. операции vMotion не будут выполнятся очень часто, забивая трафиком данный интерфейс.
Решение:
В настройке интерфейса VMkernel ставим галочку напротив опции «Fault Tolerance logging»
Установка и настройка VMware Fault Tolerance-04
2. Недостаточно ресурсов для HA
Установка и настройка VMware Fault Tolerance-05
Ошибка следующая: «Insufficient resources to satisfy configured failover level for HA»
Причина:
Для работы FT требуется работа технологии VMware High Availability. В моей среде у меня есть всего два хоста ESXi, включёнными в кластер HA, в результате в резерве у меня есть всего один сервер. Для работы FT такое решение не подходит, поэтому нужно либо добавить дополнительный хост в кластер HA, либо настроить HA на резервирование % от ресурсов, задав, например, 5%.
Решение:
Установка и настройка VMware Fault Tolerance-06
3. Тонкие (Thin) диски необходимо переконвертировать в обычные (thick)
Установка и настройка VMware Fault Tolerance-07
Ошибка: «The disk block of the virtual machine’s disks have not been fully provisioned on the file system. This is needed to support features like Fault Tolerance»
Причина:
Fault Tolerance не работает с тонкими дисками, поэтому необходимо преобразовать их в «толстые» (thick).
Решение:
Отключите тестовую виртуальную машины. Откройте хранилище с ней (Browse datastore), правой кнопкой щелкните по vmdk диску и в меню выберите “Inflate”
Вот и все!
Технология FT – это мощное решение по защите ваших бизнес-критичных приложений. Ради эксперимента я попробовал запустить пинг на машину, защищенную при помощи Fault Tolerance, и отключил первичный сервер ESXi, в результате ни одного пинга не было потеряно! Однако через интерфейс vmkernel (FT log) прошло большое количество трафика на скорости 33 Мбит/сек. Поэтому, при использовании, Fault Tolerance необходимо заранее спрогнозировать и обеспечить высокую пропускную способность на этом сегменте сети. Материал сайта Pyatilistnik.org
Phystrix
PHP-библиотека [https://github.com/odesk/phystrix Phystrix] разрабатывалась под впечатлением и по принципу Netflix Hystrix. Она создана для распределенных приложений с зависимостями.
По своей сути Phystrix – обертка для PHP-скриптов, которая изолирует их выполнение друг от друга, обеспечивая отказоустойчивость и отсутствие каскадных сбоев.
Принцип работы
При создании комманды Phystrix используется метод run(), который выполняет нужную функцию (запрос к базе данных, вызов API) и метод getFallback(), который вызывает значение по умолчанию.
Затем при вызове execute() Phystrix запускает функцию, обозначенную в run(). И в случае ошибки вызывает значение по умолчанию в getFallback(), к примеру “Функция временно недоступна”.
Для корректной работы библиотеки потребуется расширение [http://php.net/manual/en/book.apc.php APC], а для установки рекомендуется использовать [https://getcomposer.org/ Composer]:
“odesk/phystrix”: “dev-master”
}
## Будет использовать репозиторий компании на GitHub
В простейшем случае пример использования Phystrix будет выглядеть так:
**class MyCommand extends AbstractCommand**
{
protected $name;
public function __construct($name)
{
$this->name = $name;
}
/**
# Внутренний вызов функции только если запрос разрешен
# @return mixed
*/
protected function run()
{
return ‘Hello ‘ . $this->name;
}
}
## Все комманды должны использовать наследование AbstractCommand
А затем комманду можно выполнять используя метод execute():
$result = $myCommand->execute();
## Дополнительные параметры в getCommand передаются конструктору
Пример использования Phystrix
Библиотека может использоваться для множества функций, запросов HTTP, кастомных или стандартных API, а также SQL:
class GetTop extends AbstractCommand
{
protected $startDate;
public function __construct($startDate)
{
$this->startDate = $startDate;
}
protected function run()
{
# получение зависимостей DB из служебного локатора Phystrix
$db = $this->serviceLocator->get(‘db’);
$sql = ”
SELECT
U.DisplayName,
COUNT(C.ID) AS CommentCount
FROM
Users AS U
INNER JOIN Comments AS C ON U.ID = C.UserID
GROUP BY
U.DisplayName
WHERE C.time > ?
ORDER BY
COUNT(C.ID) DESC
LIMIT 1;
“;
return $db->fetchOne($sql, $this->_startDate);
}
protected function getFallback()
{
return “функция временно недоступна”;
}
}
## SQL-запрос обернут в шаблон проектирования
Ну а для запуска используется все тот же метод execute():
$phystrixFactory->getCommand(‘GetTop’, strtotime(‘-1 week’));
$topCommenterName = $getTop->execute();
## Phystrix самостоятельно проверит все зависимости
How does FT work, and what’s the history? ^
Over the years, vSphere FT has evolved. It works by continuously replicating an entire running VM from one physical server to another. The protected VM is the primary VM. Activating FT creates a duplicate VM called a secondary VM, which runs on another host.
FT continuously replicates the primary VM to the secondary VM so the secondary VM can take over at any point, thereby providing fault-tolerant protection. In case of a hardware problem on the host that runs the primary VM, it will start a new secondary VM and reestablish FT redundancy automatically.
Primary and secondary VMs
Note: This also works the other way around. In case there is a hardware problem on a machine that hosts a secondary VM, the primary VM detects it and re-creates a new secondary VM on another host within the vSphere cluster.
New Features and Support
In the opening paragraph, I used to word seldom to indicate that FT has been generally overlooked at least until vSphere 6 came along. The reason for this is due to FT’s lack of support for symmetric multiprocessing and VMs sporting more that 1GB of RAM. In fact, prior to vSphere 6.0, FT could only be enabled on VMs configured with a maximum of 1 vCPU and 1GB of RAM. Needless to say, this limitation turned out to be a show stopper since current operating systems and applications require significantly more compute resources.
NOTE: FT is not available with vSphere Essentials and Essentials Plus licensed deployments.
The improvements that vSphere 6.0 brings to FT are as follows.
-
Support for symmetric multiprocessor vms
- Max. 2 vCPUs – with vSphere Standard and Enterprise licenses
- Max. 4 vCPUs – with a vSphere Enterprise Plus license
-
Support for all types of VM disk provisioning
- Thick Provision Lazy Zeroed
- Thick Provision Eager Zeroed
- Thin Provision
- FT VMs can now be backed up using VAPD disk-only snapshots
-
Support for VMs with up to 64GB of RAM and vmdk sizes of up to 2TB
And here’s a list of VM and vCenter features that remain unsupported
- CD-ROM or floppy virtual devices backed by a physical or remote device.
- USB, Sound devices and 3D enabled Video devices
- Hot-plugging devices, I/O filters, Serial or parallel ports and NIC pass-through
- N_Port ID Virtualization (NPIV)
- Virtual Machine Communication Interface (VMCI)
- Virtual EFI (Extensible Firmware Interface) firmware
- Physical Raw Disk mappings (RDM)
- VM snapshots (remove them before enabling FT on a VM)
- Linked Clones
- Storage vMotion (moving to an alternate datastore)
- Storage-based policy management
- Virtual SAN
- Virtual Volume Datastores
- VM Component Protection (see my HA post)
What are the VMware FT limits? ^
VMware vSphere 6.5 has a few FT technical limits and there are also licensing limits. I’ll cover both.
These are the vSphere FT maximums:
- Virtual disks: 16
- Disk size: 2 TB
- Virtual CPUs per VM: 4
- RAM per FT VM: 64 GB
- FT VMs per host: 4
- Virtual CPUs per host: 8
As to VMware licensing for FT, you have access to FT with vSphere Standard, but you can only configure two vCPUs. To activate four vCPUs for your FT-protected VMs, you need to be on Enterprise Plus or vSphere with Operations Management.
VMware vSphere 6.5 Fault Tolerance licensing
Wrap-up
VMware vSphere 6.5 FT improves the integration with vSphere Distributed Resource Scheduler (DRS) and allows better placement decisions. It now ranks hosts based on available network bandwidth and datastore latency for placing the secondary VMDKs. You can use multiple port groups for FT logging traffic to add capacity to existing network bandwidth in a similar way as for Multi-NIC vMotion.
Requirements, limits, and licensing ^
- The physical CPUs need to support vMotion and hardware memory management unit (MMU) virtualization (Intel EPT or AMD RVI). They have to be Intel Sandy Bridge or later, and for AMD, they have to be AMD Bulldozer or later.
- You’ll need 10 Gb networking for FT logging; check whether the network is a low-latency one. VMware recommends dedicated networking.
Limits
- The number of VMs protected by FT per cluster: by default, this is four (but configurable with the host advanced configuration parameter: maxftvmsperhost).
- The number of virtual CPUs (vCPUs) configured within VMs protected by FT: by default, this is eight (but configurable with the host advanced configuration parameter: maxftvcpusperhost).
Licensing
The number of vCPUs supported by a single fault-tolerant VM is limited by the level of licensing you have purchased for vSphere.
FT support is as follows:
- vSphere Standard and Enterprise allow up to two vCPUs
- vSphere Enterprise Plus allows up to eight vCPUs
VM limitations
The VM needs to meet some requirements as well.
- There must be no unsupported devices attached to the VM.
- You must store the VM on shared storage, but not the VM disks (VMDKs).
- You can’t have CD-ROMs, floppies, network interface controller (NIC) passthroughs, hotplug devices, serial or parallel ports, video cards with 3D-enabled features, or 2 TB+ VMDKs.
An Overview
VMware Fault Tolerance (FT) is a process where a virtual machine, called the primary VM, replicates changes to a secondary VM created on an ESXi host other than the one hosting the primary vm. Think mirroring. In the event that the primary VM ‘s host fails, the secondary VM will take over immediately with zero interruption and downtime. Similarly, if the secondary VM’s ESXi host goes offline, a secondary VM is created on yet another host, assuming one is available. For this reason, a 3-host cluster is the recommended minimum even though FT works just as well on clusters with just 2 nodes.
Figure 1 – FT architecture (Source: www.vmware.com)
The one benefit that immediately stands out, is that Fault Tolerance raises the High Availability bar one notch higher. FT bolsters business continuity by mitigating against data loss and application downtime, something that an HA cluster on its own cannot completely deliver. FT is also useful in scenarios where expensive clustering solutions are impractical or costly to implement.
Occasionally, you may come across the term On-Demand Fault Tolerance. Since FT is a resource expensive process, you may decide to employ scripting to enable and disable FT on a schedule; just to mention one way of doing it. On-Demand FT protects business critical VMs and the corresponding business processes such as payroll applications and payroll runs against data loss and service interruption. That said, keep in mind that FT protects only at a host level. It does not provide any application level protection meaning that manual intervention is still required if a VM experiences OS and/or application failures.
Стратегия
Отказоустойчивая архитектура готова к сбоям. Приложение проектируют таким образом, чтобы при падении компоненты сами перезапускались без сбоев, потерь или ошибок.
Как пример используем скрипт для рассылки писем по базе. Не отказоустойчивая программа в случае сбоя при выполнении начнет процесс сначала, если восстановить работу.
А скрипт, написанный с учетом fault tolerance поймет, что при последнем выполнении произошел сбой и продолжит отправлять письма с пользователя, на котором произошла ошибка.
Сам скрипт выглядит так:
$restart_from = file_get_contents(‘/tmp/mail.status’);
# получаем список всех пользователей
$users = get_list(‘SELET * FROM users ORDER BY id’);
foreach ( $users as $user )
{
# пропускаем отправку, если переменная восстановления задана
if ( $restart_from && $user <= $restart_from ) continue;
# сохраняем id пользователя в файл статуса
file_put_contents(‘/tmp/mail.status’, $user);
}
# удаляем файл статуса
unlink(‘/tmp/mail.status’);
## Отсутствие файла статуса свидетельствует об успешном завершении предыдущей отправки
Еще один вариант отказоустойчивого приложения – запуск скриптов в отдельных дочерних процессах, используя многопоточность. Если появятся сбои, то процесс закончится с ошибкой, а воркер перезапустится. Для этого подойдет Gearman.
The Basic Requirements
At the time of writing, the most stringent requirement in my opinion is the 10Gbit dedicated network, which can be a hard sell to SMBs as this is something you usually find in an enterprise setting due to cost. Other than that, just ensure that the CPUs in your clusters support Hardware MMU virtualization and are vMotion ready. For Intel CPUs, anything from Sandy Bridge onwards is supported. For AMD, Bulldozer is the minimum supported micro-architecture.
On the networking side, you need to configure at least one vmkernel for Fault Tolerance Logging. At a minimum, each host should have a Gbit network card set aside for vMotion and another one for FT logging.
Figure 3 – Setting up a vmkernel for FT logging
Note: For DRS to work alongside FT, you must enable EVC. Apart from ensuring a consistent CPU profile, this allows DRS to select the optimal placement of FT protected vms.
Реклама
-
Backup
Blogs
Bug
Bugs
Certification
Cisco
Data Recovery
ESX
ESXi
Fault Tolerance
FT
HA
Hardware
Hyper-V
KB
Memory
Microsoft
Nexus
NFS
Performance
PowerCLI
PowerShell
Security
SRM
Storage
Troubleshooting
Update
vCenter
vCompute
VCP
Veeam
Video
View
VMachines
VMotion
VMware
VMworld
vNetwork
vSphere
vStorage
Бесплатно
Обучение
Производительность
Сравнение
Сравнения -
- Application vServices (270)
- AppSpeed (1)
- Cloud vServices (184)
- Data Recovery (23)
- Fault Tolerance (10)
- Infrastructure vServices (260)
- SVMotion (1)
- VCB (1)
- vCenter (218)
- vCompute (25)
- VMotion (7)
- VMsafe (5)
- VMware HA (5)
- vNetwork (30)
- vShield (4)
- vStorage (29)
-
- Этот блог пока больше не обновляется — ищем человека.
- Поддержка виртуализованных контроллеров домена Active Directory в VMware vSphere 5.1 (VM-Generation ID)
- Поздравляем лучшего российского инструктора года по продуктам VMware.
-
- Ноябрь 2013
- Январь 2013
- Октябрь 2012
- Август 2012
- Июль 2012
- Июнь 2012
- Май 2012
- Апрель 2012
- Март 2012
- Февраль 2012
- Январь 2012
- Декабрь 2011
- Ноябрь 2011
- Октябрь 2011
- Сентябрь 2011
- Август 2011
- Июль 2011
- Июнь 2011
- Май 2011
- Апрель 2011
- Март 2011
- Февраль 2011
- Январь 2011
- Декабрь 2010
- Ноябрь 2010
- Октябрь 2010
- Сентябрь 2010
- Август 2010
- Июль 2010
- Июнь 2010
- Май 2010
- Апрель 2010
- Март 2010
- Февраль 2010
- Январь 2010
- Декабрь 2009
- Ноябрь 2009
- Октябрь 2009
- Сентябрь 2009
- Август 2009
- Июль 2009
- Июнь 2009
- Май 2009
- Апрель 2009
- Март 2009
