# Refractometer Comparisons



## mathof (Mar 24, 2012)

*Note: Discussion split from Coffee Omega advert.*

In the interests of science and curiosity, I would be happy to put my recently purchased Atago up against a VST refractometer. We could run the tests in my kitchen in central London (EC1) or I could travel anywhere in reach of London Transport.

Matt


----------



## The Systemic Kid (Nov 23, 2012)

Great idea - any takers?


----------



## MWJB (Feb 28, 2012)

mathof said:


> In the interests of science and curiosity, I would be happy to put my recently purchased Atago up against a VST refractometer. We could run the tests in my kitchen in central London (EC1) or I could travel anywhere in reach of London Transport.
> 
> Matt


I have already done this. 3 paper filtered drip brews, 10 reads of each brew with both devices, test run twice, the VST has a standard deviation around 3 times smaller than the Atago (the one I used was calibrated by the vendor & cost nearly £400). There was a significant difference in readings for both tests (p=0.000). In their espresso test of refractometers, Socratic also noted the standard deviations of the VSTs used were a third of the Atago when both were used as per VST instructions.

I fail to see why Atago would deliberately publish wider tolerances if they knew their devices were higher spec.


----------



## risky (May 11, 2015)

MWJB said:


> I have already done this. 3 paper filtered drip brews, 10 reads of each brew with both devices, test run twice, the VST has a standard deviation around 3 times smaller than the Atago (the one I used was calibrated by the vendor & cost nearly £400). There was a significant difference in readings for both tests (p=0.000). In their espresso test of refractometers, Socratic also noted the standard deviations of the VSTs used were a third of the Atago when both were used as per VST instructions.
> 
> I fail to see why Atago would deliberately publish wider tolerances if they knew their devices were higher spec.


What were the deviations out of interest?


----------



## MWJB (Feb 28, 2012)

risky said:


> What were the deviations out of interest?


Test 1, sample #1, 10 reads per sample VST Lab II 0.000; Atago 0.013

Test 1 sample #2, 10 reads per sample VST Lab II 0.003; Atago 0.008

Test 1 sample #3, 10 reads per sample VST Lab II 0.003; Atago 0.014

Test 2 sample #1, 10 reads per sample VST Lab II 0.004; Atago 0.024

Test 2 sample #2, 10 reads per sample VST Lab II 0.005: Atago 0.011

Test 2 sample #3, 10 reads per sample VST Lab II 0.008; Atago 0.012


----------



## bongo (Apr 20, 2014)

What units of measurement are you posting the SD in?

I'm not familiar with using these devices, but am interested in this thread.

I'm reading this as the Atago having SD between 0.8% - 2.4%? Is this right?


----------



## MWJB (Feb 28, 2012)

bongo said:


> What units of measurement are you posting the SD in?
> 
> I'm not familiar with using these devices, but am interested in this thread.
> 
> I'm reading this as the Atago having SD between 0.8% - 2.4%? Is this right?


Standard deviations are in coffee TDS%, you are a decimal place, or two, out.

These are the Sdevs after 10 reads, so 0.024 Sdev for the Atago relates to 10 readings of the same sample, in this case the readings spanned 0.08%TDS (~2%EY difference at 17:1 brew ratio), no more than 4 reads were the same. Worst case for the VST Lab II was readings spanned 0.03%TDS over 10 reads.

Averaging out the Sdev for the 2 tests you get 0.004 for the VST & 0.014 for the Atago. To get a Sdev of 0.004 you would need 8 reads the same and 2 reads the same but only 0.01%TDS apart from the other 8.

The Atago's Sdev for Test 1 sample 3 was 0.014, these readings spanned 0.05%TDS, 4 readings the same, then 2 readings +0.01, 2 readings -0.01%TDS, 1 highest +0.02%TDS & 1 lowest -0.02%TDS.

Atago say the 3rd read "is the one". The third read for the Atago was the same as the mean for 10 reads in 1 out of 6 samples. The third read for the VST was the same as the mean in 5 out of 6 samples. The averaged Sdevs for the 1st 3 reads for the VST was 0.000 & 0.005 for test 1 & 2, compared to 0.013 & 0.020 for the Atago (3 reads isn't really enough to draw any conclusions).

What this basically means is that a t-test shows that for each test the readings between the 2 devices are significantly different & there is no doubt about this (p=0.000). This means that Coffee Omega's claim that the instruments are comparable is untrue. Neither Coffee Omega, nor Socratic Coffee have tested the "accuracy" of either instrument. Atago & VST have, that's why they publish different specs. The Atago is cheaper (though we don't know what it costs with certification, the website says it takes up to 3 weeks to get it, as I said the Atago I used was calibrated at extra charge by the seller & not off the shelf) because of this in the same way that the VST Lab II was cheaper than the VST Lab III.

Coffee Omega want to sell the Atago, fair enough, but their claims about accuracy are baseless. Precision (repeatability) can be more easily tested & that is different.


----------



## PPapa (Oct 24, 2015)

@MWJB, what alpha have you chosen for the t-test in order to claim a significant difference between two test samples? I assume it's 95% confidence, but I think it's worth mentioning anyway.

Mind you I haven't had statistics for quite some time and couldn't calculate the stdev without a cheat sheet...


----------



## MWJB (Feb 28, 2012)

PPapa said:


> @MWJB, what alpha have you chosen for the t-test in order to claim a significant difference between two test samples? I assume it's 95% confidence, but I think it's worth mentioning anyway.
> 
> Mind you I haven't had statistics for quite some time and couldn't calculate the stdev without a cheat sheet...


Why don't you insert function on XL?

Sorry. each test (30 reads).


----------



## bongo (Apr 20, 2014)

From what's been said so far I would agree with coffee omega that they are totally comparable - it also seems the accuracy statement is correct.

So far you have talked about about reliability, rather than accuracy (combination of reliability and validity & precision).

They both measure to 3pd judging by your posts so are equally precise. An ICC will give you the reliability information. You then need to test the validity. For validity purposes, you may want to look at a bland and altman plot, for instance, not just a T test. Compare the VST data to the atago data. Do you not want to visually inspect the spread of data, plotting the data sets against one another and see the overlap?

On the assumption that they both read 20% (or what ever figure you were reading). To most people the accuracy is the same. There is more variability in the atago, so it may not be as reliable - but the differences don't immediately appear meaning ful.

Just because a T-test finds a difference, doesn't mean this is a meaning full difference.


----------



## PPapa (Oct 24, 2015)

MWJB said:


> Why don't you insert function on XL?
> 
> Sorry. each test (30 reads).


My point was that I couldn't write down a function for the stdev, which isn't that difficult. (EDIT: I.e. I feel like my knowledge is quite rusty)

P.s. I used Excel for the statistics once. R/Python would be my preference anyway


----------



## Mrboots2u (May 10, 2013)

@bongo have you owned or used a refractometer and used coffee tools ?

A 2 percent deviation on extraction yield is massive ...


----------



## bongo (Apr 20, 2014)

Nope. Hence my interest in knowing more about them.

What do you mean by coffee tools?


----------



## Mrboots2u (May 10, 2013)

bongo said:


> Nope. Hence my interest in knowing more about them.
> 
> What do you mean by coffee tools?


I don't think you understand the relation between Tds and brew ratio and extraction Yield , but then post there is no difference between the measurements that a atago or vst deliver ....


----------



## bongo (Apr 20, 2014)

Mrboots2u said:


> A 2 percent deviation on extraction yield is massive ...


While I don't disagree with what you've said here, I'm not sure where this 2% has come from?

The biggest value stated so far is a standard deviation of .024%

I can't see where a difference of 2% EY is mentioned....

What am I missing?


----------



## Mrboots2u (May 10, 2013)

Mark is talking about a deviation on Tds - when making a filter this deviation could cause a difference of 2 percent in EY . You are relating the deviation in marks post to EY. This is simply not the point that is being made . Again you need some experience on use of Rome of these and measuring EY and taste to accurately comment


----------



## Mrboots2u (May 10, 2013)

bongo said:


> While I don't disagree with what you've said here, I'm not sure where this 2% has come from?
> 
> The biggest value stated so far is a standard deviation of .024%
> 
> ...


Re read marks post and have a

Look at barista hustle to understand brew ratio and tds and EY

Tds is not EY it is one variable along with others in that EY calculation


----------



## bongo (Apr 20, 2014)

Mrboots2u said:


> I don't think you understand the relation between Tds and brew ratio and extraction Yield


Sure don't... this is correct.



Mrboots2u said:


> then post there is no difference between the measurements that a atago or vst deliver ....


Going from what's been said about the tools so far, i believe this still stands true..... All that has been suggested from what i've read is the atago has a little more variance (at half the cost)...


----------



## Mrboots2u (May 10, 2013)

bongo said:


> Sure don't... this is correct.
> 
> Going from what's been said about the tools so far, i believe this still stands true..... All that has been suggested from what i've read is the atago has a little more variance (at half the cost)...


If you don't understand how EY is calculated and the variables in it and how they interact then you can't comment on how the tds deviation effects it

And therefore how it effects taste . Please do some more reading , your post is

Misleading as it is - you talk of deviations mentioned being in extraction yield . They are not - they are in tds ...tds is not extraction yields . It is one variable along with brew ratio that coffee tools used to calculate extrAction yield .


----------



## bongo (Apr 20, 2014)

While I sand corrected in influence on taste - my comments regarding the reliability & validity of the tools (which was the main theme) is correct.

I'll delete my comments on taste as you say - wrong inference there...


----------



## Mrboots2u (May 10, 2013)

bongo said:


> While I sand corrected in influence on taste - my comments regarding the reliability & validity of the tools (which was the main theme) is correct.
> 
> I'll delete my comments on taste as you say - wrong inference there...


I really would go and visit barista hustle ... Until you grasp how a recipe and strength interact in extraction yield and how they can effect taste then I would not comment on how the deviation in tds ( not extraction yield )is not important..

Barista hustle can explain this far better and easier than I can and in less

Time


----------



## GlennV (Mar 29, 2013)

Can you clarify what your claim is please, @MWJB? The t-test is appropriate for comparing the means of two distributions, but you only give standard deviations.


----------



## AndyS (May 12, 2012)

All the VST - Atago comparisons I have seen talk about comparative readings, but nothing about the build quality or ease of use. This is an unfortunate omission.

Build quality of the VST instruments is far superior. The Atago is flimsy and cheap in comparison. When you use and handle them both, this is obvious. The VST feels like a tank that will last a very long time with proper handling. I don't *know* if the VST will hold up better than the Atago, but I would bet on it doing so. [note: subjective build quality aside, they are both precision instruments that should always be handled with care]

The Atago uses a glass prism, which can be scratched. The VST LAB uses a sapphire prism which is extremely difficult to scratch. The original VST refractometers were made by Atago and included an instruction to use special lint-free tissues each time you wiped the prism clean. With the newer VST LAB series you just give a quick swipe with a standard kitchen paper towel and you're done, no worries.

The VST gives a clear, legible reading in two seconds (assuming the sample is properly cooled and filtered). The Atago sits there and "thinks" for 8-10 seconds before the reading slowly scrolls by on the screen. Presumably the Atago's slow reaction time is because it has a lower resolution sensor, and needs to take many readings before it averages them out and displays its answer. This delay isn't a big deal if you're only doing an occasional measurement, but it's annoying if you're doing a bunch of samples.

I don't know why Socratic nor anyone else ever mentions this stuff, but there are reasons why the VST costs more....

[usual disclosure: I was a beta tester for all the VST-branded refractometers (made by Atago, Reichert and MISCO). No financial interest in the company.]


----------



## MWJB (Feb 28, 2012)

GlennV said:


> Can you clarify what your claim is please, @MWJB? The t-test is appropriate for comparing the means of two distributions, but you only give standard deviations.


I gave the Sdevs because they relate more closely to precision (something that can be tested with both devices without a datum for trueness/accuracy), I mentioned the t-test because over 30 readings (10 each of 3 samples) the means & sdevs were so disparate that t-test shows the readings were different for each device.


----------



## MWJB (Feb 28, 2012)

bongo said:


> While I don't disagree with what you've said here, I'm not sure where this 2% has come from?
> 
> The biggest value stated so far is a standard deviation of .024%
> 
> ...


 @bongo one of the samples (0.024Sdev) with the Atago had a range of 0.08%TDS (strength of the coffee) in the readings, this equates to a difference in 2%EY (the extracted solubles from the coffee dose).


----------



## AndyS (May 12, 2012)

MWJB said:


> one of the samples (0.024Sdev) with the Atago had a range of 0.08%TDS (strength of the coffee) in the readings, this equates to a difference in 2%EY (the extracted solubles from the coffee dose).


I dont know what you mean by "range" but a difference of say, 1.30% to 1.38% would not produce a 2% EY difference at any realistic brew ratio.


----------



## MWJB (Feb 28, 2012)

bongo said:


> On the assumption that they both read 20% (or what ever figure you were reading). To most people the accuracy is the same.





bongo said:


> All that has been suggested from what i've read is the atago has a little more variance (at half the cost)...


That's not a logical assumption. Refractometers don't read in EY (e.g. the 18-22% box), they read in the strength of the coffee, if the numbers you use to then calculate EY (using the same convention) are different, then you get different EYs. The 2 refractometers never read "the same" in my tests, which I intended to be as fair as possible.

Refer to manufacturer's specs for accuracy specifications of calibrated/certified products.

We don't as yet know what the price for a certified unit is, so we can't make assumptions as to relative price. The retailer calibrated unit I used for the test cost rather more than the Coffee Omega price. You seem to acknowledge that the Atago costs less & has higher variation in readings, that's exactly my point. The case is the same for the VST Lab II vs VST Lab III. It's up to people what they want to do with their money but bizarre claims by resellers should be scrutinised.

It's a bit trickier in this field because not that many people understand the mechanisms involved, even if they understand the objectives, but if it was claimed that a 54mm flat burr grinder was "the same" as a 98mm flat burr grinder people would simply pull out the burrs, measure them with a ruler & call foul. The seller wouldn't bother making such a claim.


----------



## MWJB (Feb 28, 2012)

AndyS said:


> I dont know what you mean by "range" but a difference of say, 1.30% to 1.38% would not produce a 2% EY difference at any realistic brew ratio.


Apologies, you're right Andy, 1.2%EY between readings for the same device. Almost 2% difference between readings accross 2 devices.


----------



## risky (May 11, 2015)

I've split this discussion from the coffee omega advert.

This is a very interesting topic of discussion, but a commercial area sales ad wasn't the best place for it.

Mark's response to the advert has been left in the original thread where coffee omega can respond to his concerns if they wish.


----------



## mathof (Mar 24, 2012)

MWJB said:


> Test 1, sample #1, 10 reads per sample VST Lab II 0.000; Atago 0.013
> 
> Test 1 sample #2, 10 reads per sample VST Lab II 0.003; Atago 0.008
> 
> ...


I'm trying to picture what you did to take 10 reads per sample with the Atago. If you put some coffee in the well, and turn the device on, after a few seconds readings begin to scroll across the screen. This goes on for about two and a half minutes. Did you just write down the first eight readings?

Matt


----------



## doolallysquiff (Jul 26, 2014)

risky said:


> I've split this discussion from the coffee omega advert.
> 
> This is a very interesting topic of discussion, but a commercial area sales ad wasn't the best place for it.
> 
> Mark's response to the advert has been left in the original thread where coffee omega can respond to his concerns if they wish.


I doubt it, they didn't know the difference between an OPV and a pressure stat when I last contacted them.


----------



## MWJB (Feb 28, 2012)

mathof said:


> I'm trying to picture what you did to take 10 reads per sample with the Atago. If you put some coffee in the well, and turn the device on, after a few seconds readings begin to scroll across the screen. This goes on for about two and a half minutes. Did you just write down the first eight readings?
> 
> Matt


Writing down the 1st eight readings for one, and 10 for the other wouldn't have been a fair test.

For each test sample I zeroed both refractometers, brewed some drip coffee (nominally 17:1, ambient temp 22-25C for all tests), stirred it & took a sample, left to cool for a minute. Then alternately dripped drops of the sample, with a fresh pipette, into each refractometer (rotating which device I started with). After 30sec on the prism, I took 10 readings with each, alternating between them. The Atago was much slower at producing readings (though not 2 minutes per read, maybe 10-20sec, compared to the VST's couple of seconds). I waited for each Atago reading before taking the next reading with the VST. This seemed fairest as the VST samples could have been read in much less time which may have had an effect on the state of the samples. Same number of readings for both, same sample, over the same time span.


----------



## mathof (Mar 24, 2012)

Sorry, I meant 10 readings for both. Unless you cover the well of the Atago to keep the sample from evaporating, you get a series of rising readings during the 2+ minutes the device remains on. Am I right that you took the third reading that streamed by, and left it at that? And then repeated the procedure until you had recorded 10 readings per sample with both machines?

Thanks,

Matt


----------



## MWJB (Feb 28, 2012)

mathof said:


> Sorry, I meant 10 readings for both. Unless you cover the well of the Atago to keep the sample from evaporating, you get a series of rising readings during the 2+ minutes the device remains on. Am I right that you took the third reading that streamed by, and left it at that? And then repeated the procedure until you had recorded 10 readings per sample with both machines?
> 
> Thanks,
> 
> Matt


The readings did not uniformly continue to rise after the 3rd reading, in only 1 of the 6 samples was the 10th reading higher than the 3rd on the Atago. When I say I took 10 reads, I mean 10 presses of the read button - press button then write down the value that scrolled accross the screen, press button again & repeat. For 10 reads of a sample I pressed the button 10 times per device.

For just 3 presses of the button the VST Sdev dropped, for the Atago it increased, however, I wouldn't bother with just 3 reads for such a test, 10 reads would be more robust.


----------



## mathof (Mar 24, 2012)

MWJB said:


> The readings did not uniformly continue to rise after the 3rd reading, in only 1 of the 6 samples was the 10th reading higher than the 3rd on the Atago. When I say I took 10 reads, I mean 10 presses of the read button - press button then write down the value that scrolled accross the screen, press button again & repeat. For 10 reads of a sample I pressed the button 10 times per device.
> 
> For just 3 presses of the button the VST Sdev dropped, for the Atago it increased, however, I wouldn't bother with just 3 reads for such a test, 10 reads would be more robust.


Do you have any idea why the Atago continues to give changing readings for 2+ minutes after the initial one? As I understand it, the VST gives one reading only. I can never decide which of the readings that scroll across the screen after pressing the red button is meant to be "the one".


----------



## MWJB (Feb 28, 2012)

mathof said:


> Do you have any idea why the Atago continues to give changing readings for 2+ minutes after the initial one? As I understand it, the VST gives one reading only. I can never decide which of the readings that scroll across the screen after pressing the red button is meant to be "the one".


It could vary, like you say due to evaporation, or if unfiltered, due to settling of solids? I'd ignore the first 2 Atago readings and say average the next 3? I also take a few readings with the VST (not as many as 10), and go with the average, it usually doesn't change more than 0.01%TDS anyway.

I'm not really a believer in "the one".

With any of them, allowing time for samples to cool, in a clean, room temp glass/cup, before dropping on the well is a good plan (at least a minute).


----------



## mathof (Mar 24, 2012)

MWJB said:


> It could vary, like you say due to evaporation, or if unfiltered, due to settling of solids? I'd ignore the first 2 Atago readings and say average the next 3? I also take a few readings with the VST (not as many as 10), and go with the average, it usually doesn't change more than 0.01%TDS anyway.
> 
> I'm not really a believer in "the one".
> 
> With any of them, allowing time for samples to cool, in a clean, room temp glass/cup, before dropping on the well is a good plan (at least a minute).


Makes sense. Thanks.

Matt


----------



## GlennV (Mar 29, 2013)

mathof said:


> Do you have any idea why the Atago continues to give changing readings for 2+ minutes after the initial one? As I understand it, the VST gives one reading only. I can never decide which of the readings that scroll across the screen after pressing the red button is meant to be "the one".


I've checked with Socratic, and their Atagos don't do this - the displayed value doesn't change during the scrolling display.

If it's taking new measurements after it's displayed a tds value then you should be able to see this, as the led underneath the prism will flash. Can you give an example?


----------



## mathof (Mar 24, 2012)

GlennV said:


> I've checked with Socratic, and their Atagos don't do this - the displayed value doesn't change during the scrolling display.
> 
> If it's taking new measurements after it's displayed a tds value then you should be able to see this, as the led underneath the prism will flash. Can you give an example?


I've just had a close look at the operation of my Atago PAL Coffee. Here's the procedure:

1) put a cooled sample in the well

2) press GO

3) the display shows "----" for 10 seconds as the light flashes

4) first reading

5) light flashes again, briefly, and second reading comes up

6) this continues for two minutes

EXCEPT if you press GO during those two minutes "STP" (stop?) comes up on the screen, the flashes cease, and the last value keeps scrolling by (I just discovered this). NB you can turn off the machine by holding GO down for a few seconds.

I'm wondering now, if Atago has recently added this choice between continuous readings and pressing GO to stick with the latest reading.

Matt


----------



## GlennV (Mar 29, 2013)

Interesting - I wonder if it's averaging. What do you get if you try a totally stable and easy to read sample, like 2g of table sugar in 100ml of water at room temperature. Do the readings converge?


----------



## mathof (Mar 24, 2012)

GlennV said:


> Interesting - I wonder if it's averaging. What do you get if you try a totally stable and easy to read sample, like 2g of table sugar in 100ml of water at room temperature. Do the readings converge?


I'll check. There's something else it does. Along with the TDS readings, it displays a temperature reading (which I assume is that of the sample). As that temperature drops (presumably as the sample cools), the TDS goes up. In their advertisements, Atago mentions something about automatic temperature compensation. Maybe the changing readings are connected with that.

Matt


----------



## MWJB (Feb 28, 2012)

mathof said:


> I'll check. There's something else it does. Along with the TDS readings, it displays a temperature reading (which I assume is that of the sample). As that temperature drops (presumably as the sample cools), the TDS goes up. In their advertisements, Atago mentions something about automatic temperature compensation. Maybe the changing readings are connected with that.
> 
> Matt


You want to get the sample to be read, down to a stable ambient temperature, if the temp fluctuates the reading will be more inclined to change. Your sample temperature shouldn't be dropping.

Automatic Temperature Compensation (ATC) means that over the operating range, the stabilised readings in that range don't need to be compensated as they're not 20.0C (as would be the case for non ATC devices). It doesn't mean you can dump any temperature of sample in the well and get a consistent reading (as you are finding).


----------



## mathof (Mar 24, 2012)

MWJB said:


> You want to get the sample to be read, down to a stable ambient temperature, if the temp fluctuates the reading will be more inclined to change. Your sample temperature shouldn't be dropping.
> 
> Automatic Temperature Compensation (ATC) means that over the operating range, the stabilised readings in that range don't need to be compensated as they're not 20.0C (as would be the case for non ATC devices). It doesn't mean you can dump any temperature of sample in the well and get a consistent reading (as you are finding).


I see your point, and I've had better results since adopting your advice. But you may find it interesting that Atago itself claims that you can put hot coffee samples on their refractometer and get stable results:

http://atago.net/product/?l=en&f=products_coffee.html

Matt


----------



## MWJB (Feb 28, 2012)

mathof said:


> I see your point, and I've had better results since adopting your advice. But you may find it interesting that Atago itself claims that you can put hot coffee samples on their refractometer and get stable results:
> 
> http://atago.net/product/?l=en&f=products_coffee.html
> 
> Matt


I don't have the instructions for the Atago I tested handy, but they clearly said you should avoid hot samples coming into contact with the meter in case it warped the plastic housing (you could check this?). Sure, you can put a hot sample on the well & get a reading, but accuracy is tied in with temperature & the Atago has temp accuracy of +/-1C (VST +/-0.1C), so I wouldn't recommend doing so, even with the heat sink effect of the VST's stainless steel well. Atago also state 10C to 40C ambient temp & ideally you want the sample & meter to equalise for stable readings.


----------



## MWJB (Feb 28, 2012)

The claim about high temperature stability is reinforced only with regard to the 4523 Brix model in your link, not the "TDS" models.


----------



## samuellaw178 (Oct 28, 2012)

Have anyone measured the Atago when used for higher TDS range at espresso strength? If the SD doesn't scale up at higher TDS range, it should be plenty sufficient for that purpose, no? Assuming SD of 0.05% TDS - which is higher than measured here, the difference in EY would be about 0.1 EY%. Any flaw in my thinking/calculation?


----------



## froggystyle (Oct 30, 2013)

Seems the SCAE have confidence in this machine, they are set to use it in the uk brewers cup comp.....


----------



## MWJB (Feb 28, 2012)

froggystyle said:


> Seems the SCAE have confidence in this machine, they are set to use it in the uk brewers cup comp.....


That has no bearing on their specifications.


----------



## risky (May 11, 2015)

MWJB said:


> That has no bearing on their specifications.


Indeed, but it is a strong endorsement of the product which will no doubt generate sales.


----------



## MWJB (Feb 28, 2012)

risky said:


> Indeed, but it is a strong endorsement of the product which will no doubt generate sales.


That is what product placement & endorsement is supposed to do. "Strong" is a perception, based on a sponsorship offer (which is nice to see) but still doesn't affect the specifications of the units in question, nor answer any question about certification (the only UK price I can find for an Atago PAL with certification is £471 with a 2 week wait).


----------



## froggystyle (Oct 30, 2013)

MWJB said:


> That has no bearing on their specifications.


Didnt imply it did, was just posting that i had seen it on twitter, it seems Coffee Omega have done a deal to have them used, guessing for free to drum up sales.

I have no knowledge of either machine.


----------



## GlennV (Mar 29, 2013)

MWJB said:


> That is what product placement & endorsement is supposed to do. "Strong" is a perception, based on a sponsorship offer (which is nice to see) but still doesn't affect the specifications of the units in question, nor answer any question about certification (the only UK price I can find for an Atago PAL with certification is £471 with a 2 week wait).


Why are you so keen on NIST traceable certification? I don't think my Lab II had this. Is this certification just for the basic ability to measure nD, which is easy for the end user to check, or do NIST now have a standard for coffee tds?

I'm still not sure what you're claiming with respect to the means and standard deviations of your experimental measurements by the way - are you able to publish your raw data?


----------



## MWJB (Feb 28, 2012)

GlennV said:


> Why are you so keen on NIST traceable certification? I don't think my Lab II had this. Is this certification just for the basic ability to measure nD, which is easy for the end user to check, or do NIST now have a standard for coffee tds?
> 
> I'm still not sure what you're claiming with respect to the means and standard deviations of your experimental measurements by the way - are you able to publish your raw data?


I'm saying that there is a difference between the instruments, in both readings & repeatability (as would be expected, price is reflective of this, we accept this everyday with respect to all other coffee gear). The raw data published by other parties has already lead people to draw odd conclusions with respect to perceived "accuracy", I was just testing the precision (repeatability) between two units & standard deviations are more reflective of this without muddying the water with respect to perceived accuracy (the manufacturers have already done this & publish figures).


----------



## bongo (Apr 20, 2014)

Precision is not repeatability.

Reliability is repeatability.

Precision is to how small a measurement an instrument can measure. i.e., to 1cm, or 1mm etc

Accuracy is generally then these combined (google said so)

Means and standard deviations (or as you've put, SD alone) does not tell you any of this.

That's why I suggested the methods needed (statistically) to show if there is a difference.

ICC for repeatability, repeated measures anova for internal consistency, a t-test using the means for between device differences, & my preference -bland and altman to show this visually,

The problem is this thread has been muddied by talk over potential differences in measurement and how EY is then calculated - which has nothing to do with the original question you suggested you were going to answer. It is something you may be able to infer at a later date, but as yet you have not shown what the difference is between the two units (edited this as saw the result of the t-test retrospectively). Hence the raw data and 5 minutes (maybe 30mins) in excel and SPSS can answer the question.

Yawns....

Must be time for coffee.


----------



## bongo (Apr 20, 2014)

In fact, if you want the answer, send me the data and I'll answer the question (statistically) assuming the data collected is able to do this - (i'm not interested in talking about potential for types or statistical error here).

This roundabout is putting me off my latte.


----------



## bronc (Dec 10, 2012)

Some data is available here: http://socraticcoffee.com/2015/05/measuring-total-dissolved-solids-a-refractometer-comparison-part-ii/

(Raw data can be downloaded in a tab delimited text file here. As always, while we offer the data for your personal use, we kindly ask that you send a message to [email protected] before posting or presenting it in any public forum and attach appropriate acknowledgement.)


----------



## bongo (Apr 20, 2014)

cheers....i'll try and plot something pretty for later.

I can see from an early post (as had to go back and re-read some comments) a difference was shown in terms of a p value.... but i'll see if i can make the picture clearer for those who may be interested but struggle with text alone.


----------



## bongo (Apr 20, 2014)

I retract that...

@bronc having looked through that data and the site's own analysis, the only thing that wasn't done that someone in the comments points out is a reliability test (ICC) / bland & altman

While I can calculate these - i'll contact socriatic first as per their request.

Their analysis quite clearly shows no difference in measurement of the Atago and VST3 on 2 out of 3 tests (brews).

What it doesn't say is on the 3rd test, which one was 'more wrong'...it's also only data on 3 brews which is still small on the data front. The difference on the 3rd brew could simply be down to user error, not machine error.

For my own knowledge, what is classed as the 'gold standard' device for measuring TDS?


----------



## MWJB (Feb 28, 2012)

bongo said:


> Precision is not repeatability.
> 
> Reliability is repeatability.
> 
> ...


Precision is not how small an increment an instrument can measure, this is "resolution". Both devices have the same resolution but this has no direct bearing on repeatability/precision (how close measurements are grouped). Accuracy is trueness, neither Socratic nor myself have investigated this. I don't have the facilities to draw any conclusions on accuracy so would rather not imply anything beyond manufacturer's specs here.

Standard deviation relates to repeatability, anybody with a spreadsheet can enter 2 sets of 10 values centred between 1.20 & 1.30 and use a sdev function to replicate the difference that would be seen in my readings.

ANOVA uses a mean (trueness/accuracy) to establish differences, if 2 devices have a similar mean but different precision (what Socratic were testing) then no difference would be seen. However, the means could be different (but within tolerances) and precision tighter and a difference would be seen. It is not the best way to assess precision. Read the comments on the blog.


----------



## risky (May 11, 2015)

At the end of the day, @MWJB has tested both and drawn his conclusions from that testing. Obviously people are free to question/debate the results but it would probably be better if more people got hold of an Atago and put it to the test against a VST.

What @MWJB is saying echoes what I have heard some others say who have compared both devices side by side but it would be good to see more people do a direct comparison for themselves.


----------

