Here's an objective quality metric, which gives good results for
watermarked images quality assessment: Komparator.
Many thanks to Pierre Evenou who wrote the code.
FYI, the paper to cite when using Komparator is:
D. Barba, P. Le Callet, A robust quality metric for color image quality assessment, Proceedings of the IEEE International Conference on Image Processing, pp.437-440, 2003.
You'll find here:
- The C++ sources (along with a README file, explaining how to compile).
- Results data in an excel spreadsheet.
- The PPM images used in the results file.
Here's a few comments on the results, as well as some piece of advice on the *best* way to use Komparator.
In the "OverviewKomparator.xls" file, you'll find the subjective quality scores (provided by observers during subjective experiments) The mean score is written in red font. These scores are given for several watermarking algorithms (noted A1 to A10) you will also find there the objective quality scores (provided by Komparator).
Here's the way we proceed:
We compute the mean subjective quality score, as well as the mean objective quality score for each embedding algorithm (for the 5 selected input images).
The correlation coefficient computed between the subjective and objective scores proves the metric's reliability.
The 9 first watermarking algorithms comes from Peter Meerwald's web page, the 10th was proposed in "A robust image watermarking technique based on quantization noise visibility thresholds", Signal Processing, Volume 87, Issue 6, June 2007, Pages 1363-1383, F. Autrusseau and P. Le Callet.
Komparator performs better when comparing algorithms altogether.
All details are given in "Evaluation of standard watermarking techniques, Electronic Imaging 2007, Security, Steganography, and Watermarking of Multimedia Contents IX, E. Marini, F. Autrusseau, P. LeCallet, P. Campisi."
As you'll see on the spreadsheets' plots, Observers scores are given in plot #1, and objective scores are in plot #2. The lowest is the objective score, the best is the visual quality.
To summarize, if you want to assess your embedding algborithm in terms of invisibility, you'll have to launch Komparator on the 5 selected images (for every tested algorithm), get the average objective score, and put this in plot #2.
If you are having trouble with this code ask me or ask Pierre Evenou who wrote the code.
Any feedback on your results would be very appreciated, and would help a lot improving the metric. The very last version (2.0.0) of Komparator (using Cmake to compile) is right there.
This is a .zip archive including test images, an Excel spreadsheet, and a html file explaining the steps.
A Windows version is here, and the README file is there.
If you want to use Komparator with a GUI, here's the package for Windows users.
You'll have to copy all DLLs in your C:/WINDOWS/system/ folder before launching the exe file.
Please note that this source code is freely distributed under GPL licence.
Previous versions:
komparator++-1.0.0-Source
komparator++-1.0.1-Source
komparator++-1.0.2-Source
komparator++-1.2.0-Source
komparator++-1.3.0-Source
Some libraries you might need:
Furthermore, you may have to add the following two lines
in the conf/netpbm.cmake file.
Andreas Westfeld provided an R package for Komparator v2.0.0, check out here for the windows binary package and here for the Linux-like package.
You will need images.RData as well for the example below.
Here are the command lines:
require(komparator)
load("images.RData")
system.time(k <- komparator(sheep, fall, "NONE", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "INTRA", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "INTER", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "NONE", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "INTRA", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "INTER", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "NONE", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "INTRA", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "INTER", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "NONE", "CANNY")); k
system.time(k <- komparator(sheep, fall, "INTRA", "CANNY")); k
system.time(k <- komparator(sheep, fall, "INTER", "CANNY")); k
?komparator
Many thanks to Andreas for these packages.
Many thanks to Pierre Evenou who wrote the code.
FYI, the paper to cite when using Komparator is:
D. Barba, P. Le Callet, A robust quality metric for color image quality assessment, Proceedings of the IEEE International Conference on Image Processing, pp.437-440, 2003.
You'll find here:
- The C++ sources (along with a README file, explaining how to compile).
- Results data in an excel spreadsheet.
- The PPM images used in the results file.
Here's a few comments on the results, as well as some piece of advice on the *best* way to use Komparator.
In the "OverviewKomparator.xls" file, you'll find the subjective quality scores (provided by observers during subjective experiments) The mean score is written in red font. These scores are given for several watermarking algorithms (noted A1 to A10) you will also find there the objective quality scores (provided by Komparator).
Here's the way we proceed:
We compute the mean subjective quality score, as well as the mean objective quality score for each embedding algorithm (for the 5 selected input images).
The correlation coefficient computed between the subjective and objective scores proves the metric's reliability.
The 9 first watermarking algorithms comes from Peter Meerwald's web page, the 10th was proposed in "A robust image watermarking technique based on quantization noise visibility thresholds", Signal Processing, Volume 87, Issue 6, June 2007, Pages 1363-1383, F. Autrusseau and P. Le Callet.
Komparator performs better when comparing algorithms altogether.
All details are given in "Evaluation of standard watermarking techniques, Electronic Imaging 2007, Security, Steganography, and Watermarking of Multimedia Contents IX, E. Marini, F. Autrusseau, P. LeCallet, P. Campisi."
As you'll see on the spreadsheets' plots, Observers scores are given in plot #1, and objective scores are in plot #2. The lowest is the objective score, the best is the visual quality.
To summarize, if you want to assess your embedding algborithm in terms of invisibility, you'll have to launch Komparator on the 5 selected images (for every tested algorithm), get the average objective score, and put this in plot #2.
If you are having trouble with this code ask me or ask Pierre Evenou who wrote the code.
Any feedback on your results would be very appreciated, and would help a lot improving the metric. The very last version (2.0.0) of Komparator (using Cmake to compile) is right there.
This is a .zip archive including test images, an Excel spreadsheet, and a html file explaining the steps.
A Windows version is here, and the README file is there.
If you want to use Komparator with a GUI, here's the package for Windows users.
You'll have to copy all DLLs in your C:/WINDOWS/system/ folder before launching the exe file.
Please note that this source code is freely distributed under GPL licence.
Previous versions:
komparator++-1.0.0-Source
komparator++-1.0.1-Source
komparator++-1.0.2-Source
komparator++-1.2.0-Source
komparator++-1.3.0-Source
Some libraries you might need:
Furthermore, you may have to add the following two lines
- /usr/local/netpbm/include
- /usr/local/netpbm/lib
in the conf/netpbm.cmake file.
Andreas Westfeld provided an R package for Komparator v2.0.0, check out here for the windows binary package and here for the Linux-like package.
You will need images.RData as well for the example below.
Here are the command lines:
require(komparator)
load("images.RData")
system.time(k <- komparator(sheep, fall, "NONE", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "INTRA", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "INTER", "ROBERTS")); k
system.time(k <- komparator(sheep, fall, "NONE", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "INTRA", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "INTER", "SOBEL")); k
system.time(k <- komparator(sheep, fall, "NONE", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "INTRA", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "INTER", "PREWITT")); k
system.time(k <- komparator(sheep, fall, "NONE", "CANNY")); k
system.time(k <- komparator(sheep, fall, "INTRA", "CANNY")); k
system.time(k <- komparator(sheep, fall, "INTER", "CANNY")); k
?komparator
Many thanks to Andreas for these packages.
