Changes between Version 5 and Version 6 of WikiStart

Timestamp:

04/13/16 15:40:42 (8 years ago)

Author:

hfinkel

Comment:

--

WikiStart

@@ -15 +15 @@
   git clone http://git.mcs.anl.gov/genericio.git
 }}}
+
+== Output file partitions (subfiles) ==
+
+If you're running on an IBM BG/Q supercomputer, then the number of subfiles (partitions) chosen is based on the I/O nodes in an automatic way. Otherwise, by default, the GenericIO library picks the number of subfiles based on a fairly-naive hostname-based hashing scheme. This works reasonably-well on small clusters, but not on larger systems. On a larger system, you might want to set these environmental variables:
+
+{{{
+  GENERICIO_PARTITIONS_USE_NAME=0
+  GENERICIO_RANK_PARTITIONS=256
+}}}
+
+Where the number of partitions (256 above) determines the number of subfiles used. If you're using a Lustre file system, for example, an optimal number of files is:
+
+  # of files * stripe count  ~ # OSTs
+
+On Titan, for example, there are 1008 OSTs, and a default stripe count of 4, so we use approximately 256 files.