Remove redundant lockings if possible

[amityahav]

see if we could remove redundant sync lockings for better performance without creating data racing so this needs to be checked and could be with race detection go supports

[hyperioxx]

@amityahav Tried switching to use a sync.Map instead but i feel like i got worse performance from doing so. Will test again

[amityahav]

@hyperioxx as for the clients map I think we can get rid of it's locks as suggested in the other issue #97 (review)