Your idea of count(*) is incorrect. Check this :


create table t1 (col1 int);
insert into t1 values (1);
insert into t1 values (1);
insert into t1 values (2);
insert into t1 values (2);

select count(*) from t1 group by col1
(count(*))

2
2

You need to use temp table for "group by" results, or you can use
select count(unique col1) from t1

I doubt there is a standard way to stop query after N records are selected.
"first N" does not work in stored procedures.
I testes something like this :
select case when count(unique col1) > 1
then 1
else 0
end
from t1

in a hope that optimizer would stop query processing after condition "count(unique col1 ) > 1" is met. Unfortunately, it doesn't.

What you can do is
1. insert into t2 select ...group ..
2. from another session, run monitoring script
select count(*) from t2

when that select returns more then 5000, you can interrupt the first session.