For this empirical study we used the following tools with their given versions. The project PolyJIT contains a meta project that downloads and builds all necessary libraries in the given configuration automatically.
Tool | URL | Version |
---|---|---|
PolyJIT | https://github.com/PolyJIT/PolyJIT.git | 9c69b69 |
LLVM | https://llvm.org/git/llvm.git | e8afbbe |
clang | https://llvm.org/git/clang.git | 11fbce0 |
Polly | https://llvm.org/git/polly.git | 7f80afe |
Benchbuild | https://github.com/PolyJIT/benchbuild.git | 7cf4556 |
The PolyJIT project serves as a meta project that generates a source distribution with all necessary. Using this project you can specifiy all dependent versions at configure time. You need a standard unix build environment with a few additional libraries. Ubuntu 16.04 is known to work with the meta project, given that necessary libraries like libpqxx
and likwid
are available on the system. The cmake configuration will guide you through all missing dependencies.
All experiment results and plots are derived from the following raw data.
Data | File |
---|---|
Whole program (per config) | complete |
Whole program (per config #2) | runtime |
Regions (With Speedups) | region compare |
Regions (Timings) | regions |
In the following section you can find the projects involved in this empirical study. All projects are listed with their version and source url. Benchbuild will download these automatically in the given version and cache the projects in a temporary folder for future reuse.
Many projects found in the later sections do not have a corresponding name in the project list. This is caused by dynamic projects that are handled by BenchBuild, for example lnt based projects are listed as their top category SingleSourceBenchmark but not each project from the category is listed individually. However, individual projects are visible in the later stages of this document. In some cases we cannot provide a fixed version number, as it is missing from the project description.
We use the following three datasets for this analysis. The .csv files can be generated from benchbuild with a successfull run of the ijpp experiment.
$> benchbuild run -E ijpp
This runs a fresh benchbuild experiment with a new experiment ID exp_id
. The input data can be regenerated using the following command.
$> benchbuild report -E ijpp -e <exp_id>
If your experiment did not filter any projects, this will take a while. For example, for our analysis here we use exp_id=4ae29f2f-32d3-4e7c-977e-972c39d01b58
.
ijpp_all <- read_csv("data/ijpp_report_runtime_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
ijpp_report_complete <- read_csv("data/ijpp_report_complete_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
irc <- read_csv("data/ijpp_report_region_compare_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
ijpp_regions <- read_csv("data/ijpp_report_regions_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
The following table provides a complete overview over all programs that managed to fully complete the experiment ijpp.
t_per_config <- gen_t_per_config(ijpp_all)
ijpp_report_runtime <- ijpp_all[ijpp_all$group != "polybench-mod",]
ijpp_report_runtime <- ijpp_report_runtime[ijpp_report_runtime$project %in% t_per_config$Project,]
ijpp_with_baselines <- add_baseline_comparison(ijpp_report_runtime)
DT::datatable(t_per_config)
Histogram of all regionwise measurements, excluding the band between speedup values of –1.1 and 1.1. We used a binwidth of 0.5.
plot_region_histogram(irc_gb, store=FALSE, w=7, h=6, "figures/region_histogram")
Pointcloud and violin plots of the regionwise results.
plot_region_wise_points(ijpp_regions_f, store=FALSE, w=7, h=6, filename="figures/region-per-config-speedups")
plot_region_wise_violin(ijpp_regions_f, store=FALSE, w=7, h=6, filename="figures/region-per-config-vio-speedup-polly")
plot_pointcloud2D(ijpp_with_baselines, filename="figures/per-config-speedups")
plot_pointcloud2D(ijpp_good_o3, filename="figures/per-config-speedups-better-than-o3")
plot_vioplot(ijpp_with_baselines, use_polly=FALSE, filename="figures/per-config-vio-speedup-o3")
## Warning: Removed 56 rows containing non-finite values (stat_ydensity).
plot_bar_speedup(ijpp_with_baselines, use_polly=FALSE, filename="figures/per-config-bar-all-o3")
plot_speedup_boxplot(ijpp_good_o3, filename="figures/per-config-bp-speedup-o3")
Violin plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_viobar(ijpp_good_o3, use_polly=FALSE, filename="figures/vio-good-bp-per-config_o3")
Bar plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_bar_speedup(ijpp_good_o3, use_polly=FALSE, filename="figures/per-config-bar-speedup-o3")
Box plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_speedup_boxplot(ijpp_bad_o3, filename="figures/per-config-bp-slowdown-o3")
Violin plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_viobar(ijpp_bad_o3, use_polly=FALSE, filename="figures/vio-bad-bp-per-config_o3")
## Warning: Removed 56 rows containing non-finite values (stat_ydensity).
## Warning: Removed 56 rows containing non-finite values (stat_boxplot).
The following plots consider a baseline configuration of Static (-O3 -polly -parallel).
plot_pointcloud2D(ijpp_good_polly, filename="figures/per-config-speedups-better-than-polly")
plot_vioplot(ijpp_with_baselines, filename="figures/per-config-vio-speedup-polly")
## Warning: Removed 50 rows containing non-finite values (stat_ydensity).
Violin plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_viobar(ijpp_good_polly, use_polly=TRUE, filename="figures/vio-good-bp-per-config_polly")
Bar plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_bar_speedup(ijpp_good_polly, use_polly=TRUE, filename="figures/per-config-bar-speedup-polly")
Violin plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_viobar(ijpp_bad_polly, use_polly=TRUE, filename="figures/vio-bad-bp-per-config_polly")
## Warning: Removed 50 rows containing non-finite values (stat_ydensity).
## Warning: Removed 50 rows containing non-finite values (stat_boxplot).