This Linux SDK is a collection of kernel modules and OS components that can be installed over an existing OS installation to provide base support for NVDIMM-N technology. It is meant to allow early adopters and application developers to have access to NVDIMM technology and have an environment to experiment and do development.
IMPORTANT: This SDK is not for production use. Its support model is very limited and the software being provided is as-is and considered experimental.
The SDK is intended to be installed on a system previously installed with one of the following:
Note: The RPMs provided in the SDK are secure boot signed. Please refer to the instrucitons at the HPE website (here) for obtaining and importing the keys necessary to verify the signatures.
1. Use the following command to set up your yum repository.
$ sudo yum-config-manager --add-repo http://linux.hpe.com/nvdimm/download/LinuxSDK/sdk.repo
There is a known issue with yum and if you run into an error described in Redhat bug 1184912 you can work around the issue with the following commands:
$ wget http://linux.hpe.com/nvdimm/download/LinuxSDK/sdk.repo $ sudo mv sdk.repo /etc/yum.repos.d/hpenvdimmsdk.repo
2. Then run:
$ sudo yum update
3. To install the NVM library
$ sudo yum install libpmem libpmem-devel
4. To install the NVDIMM utilities:
$ sudo yum install hpe-nvdhealth
5. When all packages are installed, reboot the system.
1. Download RPMs using the website: http://linux.hpe.com/nvdimm
NOTE: Be sure to update RPM files names with the proper versions before using the commands below
2. Install the kernel RPMs:
$ sudo rpm -ihv kernel-xxx.hpoj.x86_64.rpm kernel-core-xxx.hpoj.x86_64.rpm kernel-modules-xxx.hpoj.x86_64.rpm
3. Boot the new kernel. It should be the default choice in GRUB.
4. Install the library:
$ sudo rpm -ihv libpmem-xxx.hpoj.x86_64.rpm libpmem-devel-.hpoj.x86_64.rpm
5. Install the kernel-devel RPM if needed:
$ sudo rpm -ihv kernel-devel-xxx.hpoj.x86_64.rpm
6. Install the NVDIMM utilities:
$sudo rpm -ihv hpe-nvdhealth-xxx.hpoj.x86_64.rpm
7. When all rpms are installed, reboot the system.
SDK supports the following I/O types on NVDIMM.
1. Direct access (mmap) using DAX - Experimental
2. Block device access
3. Block device access with sector atomicity using BTT
1. Create ext4 or xfs File System on an NVDIMM (example: ext4):
$ sudo mkfs -t ext4 /dev/pmem0
2. Mount it with DAX:
$ sudo mkdir /mnt/pmem0
$ sudo mount -o dax /dev/pmem0 /mnt/pmem0
Note that DAX is an experimental feature. When using DAX with ext4 or xfs, the following message will appear in /var/log/messages and on the console:
"DAX enabled. Warning: EXPERIMENTAL, use at your own risk"
At this point, any I/O to the files under /mnt/pmem0 will be performed directly to NVDIMM without going through the page caches. That is:
DAX also bypasses the block I/O stack to minimize the software overhead.
Optionally, you can use NVM libraries (ex. libpmem) to create/access files in /mnt/pmem0. The following example uses libpmem with an existing file in /mnt/pmem0 called foo where foo is at least 16 bytes long.
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <libpmem.h>
/* cc -lpmem -lpthread ... */
main()
{
int fd = open("/mnt/pmem0/foo", O_RDWR);
if (!fd) {
perror ("open");
exit(1);
}
void* pmem = pmem_map(fd);
if (!pmem) {
perror ("pmem_map");
exit(2);
}
printf("pmem_is_pmem() returns %d\n", pmem_is_pmem(pmem, 16));
/* should return 1 */
}
See libpmem(3) for more info.
Open "/dev/pmem0" with O_DIRECT option enables the DAX feature on block device access. This allows direct access to NVDIMM with the read()/write()/mmap() interfaces without having File System on.
int fd = open("/dev/pmem0", O_RDWR|O_DIRECT);
1. Create any type of File Systems on an NVDIMM (example: ext4):
$ sudo mkfs -t ext4 /dev/pmem0
2. Mount it:
$ sudo mkdir /mnt/pmem0
$ sudo mount /dev/pmem0 /mnt/pmem0
All I/Os to the files under /mnt/pmem0 will be performed through the page caches, unless an application specifies O_DIRECT in open() for the read()/write() operations. mmap() always accesses to the page caches.
Open "/dev/pmem0" allows access to NVDIMM through the page caches without File System. Open with O_DIRECT option allows bypassing the page caches for the read()/write() interfaces, but not for the mmap() interface.
Block Translation Table (BTT) assures sector atomicity when updating data on NVDIMM. For convenience, SDK provides BTT commands for binding/unbinding BTT to/from NVDIMM. BTT supports 512 and 4096 sector sizes. btt-bind(8) sets 4096 to the sector size by default. The use of 512 will lead higher overhead. Please see man pages, btt-bind(8) and btt-unbind(8), for the command syntax.
1. As root, bind the BTT driver to the specific NVDIMM device, in this example pmem0:
Note, this step overwrites the data in pmem0 and creates a /dev/pmem0s device. This BTT binding persists across reboots.
# btt-bind 0
2. Create any type of File Systems on an NVDIMM (example: ext4):
# mkfs -t ext4 /dev/pmem0s
3. Mount it:
# mkdir /mnt/pmem0
# mount /dev/pmem0s /mnt/pmem0
For unbinding BTT from pmem0:
# umount /mnt/pmem0
# btt-unbind 0
# mkfs -t ext4 /dev/pmem0
Note, mkfs is only necessary to persist the unbinding by clearing the BTT metadata which is at the end of the pmem device. This steps overwrites the data in pmem0s. Other methods of clearing the metadata, such as copying zeros to the device, will also work.
The step to bind/unbind BTT is the same as the File System Access above. Open "/dev/pmem0s" allows access to NVDIMM through the page caches without File System. Open with O_DIRECT option allows bypassing the page caches for the read()/write() interfaces, but not for the mmap() interfaces.
1. The following warning message is logged in dmesg and /var/log/messages. This is harmless and can be ignored.
nd_pmem namespace0.0: unable to guarantee persistence of writes
2. When "intel_iommu=on" is specified in the kernel boot options, a crash dump may not be captured after a crash. Do not use this kernel option when kdump is configured.
3. Explicitly size crashkernel with "crashkernel=256M,low crashkernel=256M,high".
4. iostat(1) does not report I/O statistics information on NVDIMM devices when DAX is used.
5. msync(2) does not flush processor caches associated with the target mmap'd file. When DAX is used, an application program needs to flush processor caches by using 'clflush' instruction or NVM library interfaces in order to persist updates to an mmap'd file. Non-temporary store instructions can be used as an alternative to avoid the flushing.
Verify that you are using up-to-date ROM, CPLD, and iLO firmwares.
1. Verify that your NVDIMMs are installed correctly and that you have followed the population recommendations.
2. Verify that NVDIMM functionality is enabled in RBSU.
3. Verify that you see some NVDIMM-related messages output to the console towards the end of POST.
4. Verify that iLO reports the NVDIMMs as healthy".
5. Verify that you are running a kernel with NVDIMM enablement. When using the Orange Linux SDK, then 'uname -r' should report a version string that includes "hpoj", for example: "4.2.0-168.6.hpoj.x86_64".
Sometimes, after booting, Linux may complain (when mounting, for example), that a /dev/pmemX is read-only. You can also see this in the RO column of lsblk output.
Check the NVDIMM status flags set in the ACPI NFIT:
$ grep -H . /sys/bus/nd/devices/*/nfit/flags
It is normal for no flags to be set and for the grep command to display no output. If you see a flag of "not_armed", then this means that the NVDIMM is not armed for backup and so Linux marks it read-only. NVDIMMs will be marked not_armed if ROM is not able to arm them for some reason, such as an erase error. Check the IML or iLO log for errors.
Often this means that NVDIMM functionality is not enabled in RBSU; verify that NVDIMM is enabled in RBSU. Sometimes upgrading your ROM will cause NVDIMM to become disabled.
Report major issues through your beta program representative. They will work with the Linux SDK team
to try to find you and answer. When
receiving support you may be asked to run the NVDIMM health tool (hpe-nvdhealth)
included in the Linux SDK package and provide the output to support
personal.