eBACS: ECRYPT Benchmarking of Cryptographic Systems
|How to submit new software:||Tips||hash||stream||aead||dh||kem||encrypt||sign|
|List of primitives measured:||sha3||hash||stream||caesar||aead||dh||kem||encrypt||sign|
|Measurements indexed by machine:||sha3||hash||stream||caesar||aead||dh||kem||encrypt||sign|
Download and unpack SUPERCOP on your development machine, and switch to the top supercop-... directory.
During development you won't want to try every possible compiler option. Edit okcompilers/c to remove every line after the first. Currently the first line is gcc -march=native -mtune=native -O3 -fomit-frame-pointer -fwrapv. If this doesn't work on your machine, try just gcc -O3 -fomit-frame-pointer.
Similarly edit okcompilers/cpp.
Inside the appropriate directory for the operation that you want to implement (e.g., crypto_hash or crypto_kem or crypto_sign), create a new subdirectory for your primitive: e.g., crypto_sign/square2048. The subdirectory name should consist solely of digits (0123456789) and lowercase ASCII letters (abcdefghijklmnopqrstuvwxyz); please omit dashes, dots, slashes, and other punctuation marks.
Try compiling and testing an existing primitive to see what a successful test looks like:
./do-part used ./do-part crypto_sign ed25519
./do-part used takes about 20 minutes on a typical machine (3GHz Skylake using one core) to compile various subroutines: basic modules that are required for the benchmarking framework; the gmp, ntl, and cryptopp libraries; and cryptographic subroutines marked as "used", such as crypto_hash_sha512. (For comparison, compiling and measuring everything with one compiler option takes roughly a day on one core.) If this number of minutes is too long to wait, you can save most of the time by doing something like
./do-part init ./do-part crypto_verify 32 ./do-part crypto_hash sha512 ./do-part crypto_stream chacha20 ./do-part crypto_rng chacha20 ./do-part crypto_sign ed25519
but this requires you to know the whole chain of relevant subroutines.
Running ./do-part crypto_sign ed25519 takes only 10 seconds. Also look at the resulting data file.
Now try testing your own primitive:
./do-part crypto_sign square2048
This will run instantaneously but will produce a "measure: not found" error message, since you don't have any implementations.
Make a crypto_sign/square2048/ref directory with empty files api.h and sign.c, and try the test again. You'll again see "measure: not found", and more error messages coming from the fact that api.h doesn't define CRYPTO_PUBLICKEYBYTES etc.
The basic development loop at this point is to edit files in crypto_sign/square2048/ref, run ./do-part crypto_sign square2048 again, and repeat until you're happy with the results. Then put a tarball of crypto_sign/square2048 on the web and submit the URL to the eBATS mailing list.
To generate random bytes in public-key software, #include "randombytes.h" and then call randombytes(buf,len). randombytes is a strong random-number generator (RNG) provided by SUPERCOP.
Don't use the standard rand or random functions: they aren't cryptographically strong. If possible, don't use RNGs from other libraries, such as the OpenSSL RNG: SUPERCOP's automatic testing (see "Checksums" below) relies on all randomness coming from randombytes.
The benchmarks allow cryptographic implementations to call subroutines that are listed earlier in OPERATIONS and marked as "used". For example, your crypto_sign_square2048 software can #include "crypto_hash_sha256.h" and then call crypto_hash_sha256().
You should use crypto_verify for constant-time comparisons, for example, rather than writing your own comparison code (or calling non-constant-time libraries such as memcmp).
You can also write your own subroutines: for example, you might write a new crypto_stream_aey to be used inside a new crypto_aead_aey.
You need to create crypto_stream/aey/used to mark the subroutine as "used". You also need to be careful to have the crypto_stream/aey/ref implementation stay within the crypto_stream_aey_* namespace, preferably the crypto_stream_aey_ref_* namespace, so that it doesn't bump into other subroutines.
One effect of including "crypto_stream.h" is that crypto_stream is automatically renamed crypto_stream_aey_ref and crypto_stream_xor is automatically renamed crypto_stream_aey_ref_xor. The required definitions in api.h are also renamed automatically. However, you'll have to be careful for other functions that you define as part of the crypto_stream implementation, and for other names that you define outside functions. It's okay to use short names for static constants.
SUPERCOP automatically generates a deterministic list of inputs for your software, and hashes together the outputs into two checksums: a "small" checksum meant to very quickly weed out most bad implementations, and a "big" checksum that tries more inputs.
These checksums appear (separated by slashes) on "try" lines in the SUPERCOP database. The word after the checksums is "ok" or "fails" or "unknown"; "ok" means that the checksums match the files checksumsmall and checksumbig in the primitive directory, "fails" means that the checksums don't match (and then SUPERCOP will discard the implementation without benchmarking it), and "unknown" means that the files are absent.
Of course it's possible for a bug to slip past both checksums, but many bugs are caught by checksums, so you should include checksums.
If you write an implementation using, e.g., AVX vector intrinsics, then you should create a file architectures in the implementation directory with two lines: amd64 and x86. This saves time in benchmarking: it tells SUPERCOP to skip trying to compile the implementation on, e.g., ARM.
Similarly, if you write an implementation that's ARM-specific, you should create a file architectures with three lines: aarch64, armeabi, and arm.
There's a file crypto_hash/sha1/description saying "SHA-1 with 160-bit output", and a file crypto_hash/sha1/designers saying "NSA". This information goes into the online list of primitives. Typically designers is a list of names, one per line.
There's also a file crypto_hash/sha1/openssl/implementors saying "Daniel J. Bernstein (wrapper around OpenSSL)". This information goes into the online list of implementations. Typically implementors is a list of names, one per line.
If these files don't exist, they're treated as blank.
You are encouraged to include additional files such as README or documentation.pdf with references, intellectual-property information, descriptions of the software, etc. These files do not interact with SUPERCOP's benchmarking but are often of interest for human readers.
In particular, you are encouraged to clearly specify one of the following levels of copyright protection:
You are also encouraged to clearly specify one of the following levels of patent protection:
No matter what the copyright status is, and no matter what the patent status is, all software included in SUPERCOP will be distributed to the public to ensure verifiability of benchmark results. You must ensure before submission that publication is legal.