@Graque said in #20:
> I didn't read the papers, so sorry if this is an elementary question, but it looks like the Amsterdam Chess Test was composed intuitively at first (i.e. the authors just threw in what they thought might be related). So there's guarantee that this is the best possible test, or even a particularly good one right?
>
> Looking at the correlations, it seems the tactics-related components were much more correlated to Elo. Is it possible that any validity is due to the choose-a-move/tactics components? My intuition (based on nothing) is that the recall, verbal knowledge, and motivation pieces aren't useful.
>
> If it basically comes down to tactics, then perhaps any tactics trainer (like puzzle storm) would be as good as the ACT.
It's a very good question and you're absolutely right. This is something I always emphasize to my students: Scale development is hard and you always end up starting by making guesses about stuff that seems reasonable to include. That always means there is a good chance you added in elements that aren't especially useful or missed other things that would have been apt.
The authors definitely end up saying that the choose-a-move and predict-a-move subscales are doing most of the heavy lifting. Those other tasks do end up covering some unique variance in their model fits (which is good) but it's not as much. Again, always a challenge - how many tasks (and which ones) do you add to try and account for as much variability as you can?
It would definitely be neat to take data from lichess to see how things like puzzle storm or other easy-to-administer tests do compared to the ACT. This is a case where there is so much data and so many users, I feel like you could tinker with things very quickly.
> I didn't read the papers, so sorry if this is an elementary question, but it looks like the Amsterdam Chess Test was composed intuitively at first (i.e. the authors just threw in what they thought might be related). So there's guarantee that this is the best possible test, or even a particularly good one right?
>
> Looking at the correlations, it seems the tactics-related components were much more correlated to Elo. Is it possible that any validity is due to the choose-a-move/tactics components? My intuition (based on nothing) is that the recall, verbal knowledge, and motivation pieces aren't useful.
>
> If it basically comes down to tactics, then perhaps any tactics trainer (like puzzle storm) would be as good as the ACT.
It's a very good question and you're absolutely right. This is something I always emphasize to my students: Scale development is hard and you always end up starting by making guesses about stuff that seems reasonable to include. That always means there is a good chance you added in elements that aren't especially useful or missed other things that would have been apt.
The authors definitely end up saying that the choose-a-move and predict-a-move subscales are doing most of the heavy lifting. Those other tasks do end up covering some unique variance in their model fits (which is good) but it's not as much. Again, always a challenge - how many tasks (and which ones) do you add to try and account for as much variability as you can?
It would definitely be neat to take data from lichess to see how things like puzzle storm or other easy-to-administer tests do compared to the ACT. This is a case where there is so much data and so many users, I feel like you could tinker with things very quickly.