What is benchmarking
Benchmarking is an important tool we use to access the relative performance of a software or specific component of a software, normally by running a number of standard tests and trials against it.
Issues we need to notice for benchmarking
- Repeatability
- ability to produce same results reliably (same data, multiple runs)
- Test conditions
- make sure we use same testing data, platform and environment
- Data size and & data composition
- use enough data to get a decent run, especially using sampling sampling profilers.
- use data representative of real-world, ensure code path coverage.
- Caching - first run code will load up the cache, subsequent run will faster
- run more than one time
- or flush your cache everytime
- Avoid including scaffolding/unrelated functions in benchmarking
- benchmark the overhead seperately or substract it
- benchmark only important pieces