wiki:WikiStart
Last modified 5 years ago Last modified on 04/17/20 10:39:25

Moved to ​xgitlab hpcrc64 wiki

ALCF's High-Performance CRC64 Library

Description

This library implements a high-performance computation of the 64-bit cyclic-redundancy check (CRC) of arbitrary input data. This implementation requires only a C99 compiler, but has been tuned for the IBM BG/Q supercomputer, and can take advantage of a compiler's OpenMP support to compute the CRC in parallel on multiple cores.

Computing and storing or transmitting the CRC code of data while it is written to disk or transmitted over the network, and then using it to verify the integrity of the data upon read-in or receipt, is a powerful way to guard against data corruption. While modern disks and networks are highly reliable, the rate of corruption from malfunctioning software, hardware and external physical processes remains significant (especially for large data sets).

This implementation uses the ECMA-182 polynomial with -1 initialization, and computes the bit-reversed CRC.

Code

You can obtain the latest version of the source code directly from the public git repository: 

  git clone http://git.mcs.anl.gov/hpcrc64.git

or you can download the latest release:

Interface

This library provides the following functions for general-purpose use. For most applications, only crc64_omp (and possibly crc64_invert) will be of use:

Calculate the CRC64 of the provided buffer (in serial). 

  uint64_t crc64(const void *input, size_t nbytes);

Calculate the CRC64 of the provided buffer, in parallel if possible (using OpenMP). 

  uint64_t crc64_omp(const void *input, size_t nbytes);

Calculate the eight 'check bytes' for the provided CRC64. If these bytes are appended to the original buffer with a CRC64 value 'cs', then the new total CRC64 of the buffer with the check bytes appended should be -1. The check bytes are written to the location specified by the 'check_bytes' pointer. This pointer need not be aligned. 

   void crc64_invert(uint64_t cs, void *check_bytes);

Given the CRC64 of the first part of a buffer, and the CRC64 and length of the second part of a buffer, calculate the CRC64 of the complete buffer. 

  uint64_t crc64_combine(uint64_t cs1, uint64_t cs2, size_t nbytes2);

References

Useful references on CRCs: