Support for benchmarking memory usage, etc.

[chrish42]

This is definitely not in scope is hyperfine's goal is to be just a replacement for time, but the thing I most wish for when using time to benchmark programs is if it could measure other relevant program execution statistics, like memory usage. After all, optimization (which is what benchmarking is driving) is always about tradeoffs in some way. But it's kinda hard to know if you're making the right tradeoffs for your project if you're measuring only one thing (here, execution speed).

Would something measuring memory usage, etc. that be in scope for hyperfine a a benchmarking tool?

[sharkdp]

Thank you very much for your feedback.

This are valuable points, but I don't have any plans to extend this to memory profiling/benchmarking. I think this would turn hyperfine into a much more complicated program (code-wise an possibly usage-wise).

That being said, I'm open to discuss this.

[cauebs]

I'm not sure of what @chrish42 had in mind, but peak memory usage is an interesting information. I'm not sure how to implement it, though. If someone gives me some pointers, I might give it a go.

[sharkdp]

I tend to think this better left to (far more advanced) tools like valgrind.

[sharkdp]

I'm going to close this, as I don't have any plans to include memory profiling.

[lu-zero]

Since we use getrusage() the maximum resident set would come for free though:

rusage.ru_maxrss

I can try to bake a patch if adding this would be acceptable.

[sharkdp]

It's not that easy.

• We do not run the benchmarked process directly, but through a shell. rusuage would probably show the memory usage of both the shell process and the benchmarked subprocess. We would probably have to calibrate our memory measurement by running the shell without the subprocess first (similar to what we do for timing measurements).
• hyperfine is a cross-platform tool. rusage only works on Unix.
• To what extent would memory usage be supported as an additional metric? Do we want to build averages, stddev, etc. like we do for the time measurements? Do we run the outlier detection on the memory usage samples as well?
• How would this additional information be displayed in hyperfines terminal output?
• How would this additional information be represented in the various export formats?

[lu-zero]

It's not that easy.

* We do not run the benchmarked process directly, but through a shell. `rusuage` would probably show the memory usage of both the shell process and the benchmarked subprocess. We would probably have to calibrate our memory measurement by running the shell without the subprocess first (similar to what we do for timing measurements).

* `hyperfine` is a cross-platform tool. `rusage` only works on Unix.

The performance tools for windows have the same capabilities

* To what extent would memory usage be supported as an additional metric? Do we want to build averages, stddev, etc. like we do for the time measurements? Do we run the outlier detection on the memory usage samples as well?

It would be great to have.

* How would this additional information be displayed in `hyperfines` terminal output?

I would add another (optional) line next to the timings.

* How would this additional information be represented in the various export formats?

I hadn't thought about this, probably a second set of files would be the easiest.

[korya]

It would be great to be able to optionally get peak RAM consumption.

[sharkdp]

I'm inclined to revisit this idea. Mostly because we now have the option of running commands without an intermediate shell (-N/--shell=none). I am proposing the following stripped-down feature as a start:

• Limit this functionality to cases where --shell=none is set.
• Measure peak RAM usage using getrusage on Linux. If we do not find a working version for Windows, limit this feature to Linux only for now.
• Do not display peak RAM usage on the terminal. Only add it to the JSON export.

See #321 for an earlier implementation of this feature.

[mcandre]

Valgrind does not support Windows or macOS.

I would love for a portable tool like hyperfine to feature peak memory usage!

The current workaround involves using /usr/bin/time with either -v or -l flags. Annoyingly, the flags are mutually exclusive, depending on the particular UNIX implementation. Those hacks are doable for WSL and certain Cygwin-like environments, but there are very few equivalents for native Windows.

[lskatz]

I used that /usr/bin/time hack and made something rudamentary that might work for me. I wanted to leave it here for anyone to use or improve on.

# Run hyperfine with output printed to the screen.
# In this example, we're just running it at 5 times minimum.
$ hyperfine --show-output -m 5 \
  `# make a command with just ls` \
  -n 'ls' \
  `# run the time command and redirect ls output to /dev/null. But then redirect time output to stdout.` \
  '/usr/bin/time ls -lh > /dev/null' 2>&1 | \
  perl -lane '
    # capture any maxresident integers
    if(/(\d+)maxresident/){
      # add the maxresident to time and increment the count for later average
      $time += $1; 
      $num++
    }; 
    # If we see hyperfine output then print it untouched
    if(/ Time| Range/){
      print;
    } 
    # When hyperfine is done, then average the memory and print it
    END{
      $avg = int($time/$num); 
      print "Avg memory: ${avg}k";
    }'
  Time (mean ± σ):      10.5 ms ±   4.7 ms    [User: 2.0 ms, System: 7.9 ms]
  Range (min … max):     6.0 ms …  63.6 ms    144 runs
Avg memory: 1441k

[DrBlury]

I'd be interested in this too

[smartmic]

I came across this thread because I used hyperfine several times but now also need the peak memory usage for certain commands.
I found another tool which is comparable to hyperfind and serves me well (Unix only): multitime.

Maybe it's a good alternative for some (like me) or serves as inspiration.

[juarezr]

Related:

• poop: Performance Optimizer Observation Platform

[lu-zero]

In order to support the performance counters like poop does but keeping the tool cross platform we'd need to see how they counters are accessed over the various arches, probably dtrace is the tool that is already doing that but feels quite complex.

On Linux we could access the perf counters either via perf-event2, on macos and windows we could look at what samply is doing and maybe try to use the smallest subset of it.